[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473736289 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003167] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008773] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012064] LSM: Security Framework initializing [ 0.014071] Yama: becoming mindful. [ 0.015046] SELinux: Initializing. [ 0.016106] *** VALIDATE selinux *** [ 0.026470] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031703] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033031] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036028] *** VALIDATE tmpfs *** [ 0.037465] *** VALIDATE proc *** [ 0.038255] *** VALIDATE cgroup *** [ 0.039014] *** VALIDATE cgroup2 *** [ 0.040305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043031] Spectre V2 : User space: Vulnerable [ 0.044011] Speculative Store Bypass: Vulnerable [ 0.048058] debug: unmapping init [mem 0xffffffffa9659000-0xffffffffa9660fff] [ 0.050189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052020] ... version: 2 [ 0.053043] ... bit width: 48 [ 0.054014] ... generic registers: 4 [ 0.055015] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057017] ... fixed-purpose events: 3 [ 0.058016] ... event mask: 000000070000000f [ 0.059323] rcu: Hierarchical SRCU implementation. [ 0.061707] smp: Bringing up secondary CPUs ... [ 0.062476] x86: Booting SMP configuration: [ 0.063021] .... node #0, CPUs: #1 #2 #3 [ 0.066334] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137336] node 0 deferred pages initialised in 66ms [ 0.140268] devtmpfs: initialized [ 0.142276] x86/mm: Memory block size: 128MB [ 0.144918] gcov: version magic: 0x41383552 [ 0.148304] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152129] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154389] pinctrl core: initialized pinctrl subsystem [ 0.156154] [ 0.156720] ************************************************************* [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.164015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167015] ** ** [ 0.169011] ** This means that this kernel is built to expose internal ** [ 0.172015] ** IOMMU data structures, which may compromise security on ** [ 0.174011] ** your system. ** [ 0.176012] ** ** [ 0.179013] ** If you see this message and you are not debugging the ** [ 0.181009] ** kernel, report this immediately to your vendor! ** [ 0.184015] ** ** [ 0.186012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189012] ************************************************************* [ 0.191742] NET: Registered protocol family 16 [ 0.193410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205028] cpuidle: using governor menu [ 0.206753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209425] PCI: Using configuration type 1 for base access [ 0.212136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228146] cryptd: max_cpu_qlen set to 1000 [ 0.230274] ACPI: Added _OSI(Module Device) [ 0.232018] ACPI: Added _OSI(Processor Device) [ 0.233036] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246426] ACPI: Interpreter enabled [ 0.248067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249015] ACPI: Using IOAPIC for interrupt routing [ 0.251127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255403] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265203] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282451] acpiphp: Slot [2] registered [ 0.284202] acpiphp: Slot [5] registered [ 0.286176] acpiphp: Slot [6] registered [ 0.288098] acpiphp: Slot [3] registered [ 0.289114] acpiphp: Slot [4] registered [ 0.291151] acpiphp: Slot [7] registered [ 0.293105] acpiphp: Slot [8] registered [ 0.295116] acpiphp: Slot [9] registered [ 0.296153] acpiphp: Slot [10] registered [ 0.298220] acpiphp: Slot [11] registered [ 0.300120] acpiphp: Slot [12] registered [ 0.302088] acpiphp: Slot [13] registered [ 0.303112] acpiphp: Slot [14] registered [ 0.304104] acpiphp: Slot [15] registered [ 0.306087] acpiphp: Slot [16] registered [ 0.307104] acpiphp: Slot [17] registered [ 0.309157] acpiphp: Slot [18] registered [ 0.311126] acpiphp: Slot [19] registered [ 0.312108] acpiphp: Slot [20] registered [ 0.314108] acpiphp: Slot [21] registered [ 0.315117] acpiphp: Slot [22] registered [ 0.317102] acpiphp: Slot [23] registered [ 0.318109] acpiphp: Slot [24] registered [ 0.320169] acpiphp: Slot [25] registered [ 0.321059] acpiphp: Slot [26] registered [ 0.323158] acpiphp: Slot [27] registered [ 0.325121] acpiphp: Slot [28] registered [ 0.327148] acpiphp: Slot [29] registered [ 0.328106] acpiphp: Slot [30] registered [ 0.330133] acpiphp: Slot [31] registered [ 0.332066] PCI host bridge to bus 0000:00 [ 0.333023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353104] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358381] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365579] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.370057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380842] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384809] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388023] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.395726] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454567] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460958] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480921] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490385] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.493461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.496457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.498529] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.501274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.507212] iommu: Default domain type: Passthrough [ 0.509660] SCSI subsystem initialized [ 0.511139] ACPI: bus type USB registered [ 0.513232] usbcore: registered new interface driver usbfs [ 0.515105] usbcore: registered new interface driver hub [ 0.517094] usbcore: registered new device driver usb [ 0.519203] pps_core: LinuxPPS API ver. 1 registered [ 0.521014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.524109] PTP clock support registered [ 0.526036] EDAC MC: Ver: 3.0.0 [ 0.528168] PCI: Using ACPI for IRQ routing [ 0.529516] NetLabel: Initializing [ 0.530010] NetLabel: domain hash size = 128 [ 0.532011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.534089] NetLabel: unlabeled traffic allowed by default [ 0.538070] vgaarb: loaded [ 0.540370] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.542025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.551860] clocksource: Switched to clocksource kvm-clock [ 0.663174] VFS: Disk quotas dquot_6.6.0 [ 0.666377] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.669297] *** VALIDATE ramfs *** [ 0.670770] *** VALIDATE hugetlbfs *** [ 0.672555] pnp: PnP ACPI init [ 0.675287] pnp: PnP ACPI: found 6 devices [ 0.694069] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.699567] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.701993] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.704336] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.707300] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710174] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.714770] NET: Registered protocol family 2 [ 0.717683] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.723118] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.726833] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.732939] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.737184] TCP: Hash tables configured (established 65536 bind 65536) [ 0.740557] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.744448] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747373] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.750856] NET: Registered protocol family 1 [ 0.754544] RPC: Registered named UNIX socket transport module. [ 0.756452] RPC: Registered udp transport module. [ 0.757946] RPC: Registered tcp transport module. [ 0.759887] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.762351] NET: Registered protocol family 44 [ 0.764146] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.766946] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.769345] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.771993] PCI: CLS 0 bytes, default 64 [ 0.773983] Unpacking initramfs... [ 2.230076] debug: unmapping init [mem 0xffff9458bcc64000-0xffff9458bffcffff] [ 2.234344] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.237419] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.240967] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.753313] Initialise system trusted keyrings [ 2.755341] Key type blacklist registered [ 2.758791] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.770522] zbud: loaded [ 2.773711] *** VALIDATE nfs *** [ 2.775236] *** VALIDATE nfs4 *** [ 2.776980] pstore: using deflate compression [ 2.780457] Platform Keyring initialized [ 2.880688] NET: Registered protocol family 38 [ 2.882359] Key type asymmetric registered [ 2.883737] Asymmetric key parser 'x509' registered [ 2.885788] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.890039] io scheduler mq-deadline registered [ 2.891202] io scheduler kyber registered [ 2.892384] io scheduler bfq registered [ 2.893957] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.896796] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.899669] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.902065] ACPI: Power Button [PWRF] [ 2.907830] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.914693] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.923584] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.951398] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.981425] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.987461] Non-volatile memory driver v1.3 [ 2.990090] Linux agpgart interface v0.103 [ 3.023687] virtio_blk virtio1: [vda] 133944 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.027367] vda: detected capacity change from 0 to 68579328 [ 3.053476] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.056768] vdb: detected capacity change from 0 to 1073741824 [ 3.066757] libphy: Fixed MDIO Bus: probed [ 3.071780] usbcore: registered new interface driver usbserial_generic [ 3.073624] usbserial: USB Serial support registered for generic [ 3.075668] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.079129] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081094] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.083906] mousedev: PS/2 mouse device common for all mice [ 3.087245] rtc_cmos 00:05: RTC can wake from S4 [ 3.090638] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.090727] rtc_cmos 00:05: registered as rtc0 [ 3.097324] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.097329] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.097379] intel_pstate: CPU model not supported [ 3.103076] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.109825] hid: raw HID events driver (C) Jiri Kosina [ 3.112404] usbcore: registered new interface driver usbhid [ 3.114356] usbhid: USB HID core driver [ 3.115994] drop_monitor: Initializing network drop monitor service [ 3.118731] Initializing XFRM netlink socket [ 3.121628] NET: Registered protocol family 10 [ 3.125289] Segment Routing with IPv6 [ 3.126727] NET: Registered protocol family 17 [ 3.129024] mpls_gso: MPLS GSO support [ 3.134726] RAS: Correctable Errors collector initialized. [ 3.137151] AVX version of gcm_enc/dec engaged. [ 3.139130] AES CTR mode by8 optimization enabled [ 3.241291] sched_clock: Marking stable (3241265884, 0)->(4179128540, -937862656) [ 3.246454] registered taskstats version 1 [ 3.248395] Loading compiled-in X.509 certificates [ 3.250106] zswap: loaded using pool lzo/zbud [ 3.274612] Key type big_key registered [ 3.287070] Key type encrypted registered [ 3.288538] ima: No TPM chip found, activating TPM-bypass! [ 3.290281] ima: Allocated hash algorithm: sha1 [ 3.292107] ima: No architecture policies found [ 3.294652] evm: Initialising EVM extended attributes: [ 3.297287] evm: security.selinux [ 3.299134] evm: security.ima [ 3.300807] evm: security.capability [ 3.302560] evm: HMAC attrs: 0x1 [ 3.307192] rtc_cmos 00:05: setting system clock to 2026-04-01 11:11:08 UTC (1775041868) [ 3.315443] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 3.318868] debug: unmapping init [mem 0xffffffffa9382000-0xffffffffa9658fff] [ 3.328174] Write protecting the kernel read-only data: 28672k [ 3.331871] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 3.335246] debug: unmapping init [mem 0xffffffffa8314000-0xffffffffa83fffff] [ 3.372092] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.381909] systemd[1]: Detected virtualization kvm. [ 3.383932] systemd[1]: Detected architecture x86-64. [ 3.386065] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.415296] systemd[1]: No hostname configured. [ 3.417162] systemd[1]: Set hostname to . [ 3.419434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.422290] systemd[1]: Initializing machine ID from random generator. [ 3.463273] random: ln: uninitialized urandom read (6 bytes read) [ 3.561648] random: systemd: uninitialized urandom read (16 bytes read) [ 3.564433] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.569388] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.573324] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.258609] device-mapper: uevent: version 1.0.3 [ 4.261759] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.072966] virtio_net virtio0 ens2: renamed from eth0 [ 5.154767] scsi host0: ata_piix [ 5.212232] scsi host1: ata_piix [ 5.213800] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.217640] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.530865] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.811310] random: crng init done [ 9.812887] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.334830] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.570970] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.858921] SELinux: Disabled at runtime. [ 11.920759] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.928670] systemd[1]: Detected virtualization kvm. [ 11.931053] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.416554] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.419944] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.425531] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.428553] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.432100] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.444315] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.450564] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ 12.592870] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.913787] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.329103] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.390055] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.472953] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.495079] EDAC sbridge: Ver: 1.1.2 [ 14.735436] Key type dns_resolver registered [ 15.055912] NFS: Registering the id_resolver key type [ 15.058070] Key type id_resolver registered [ 15.059624] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-client login: [ 45.189494] libcfs: loading out-of-tree module taints kernel. [ 45.224218] Key type ._llcrypt registered [ 45.233802] Key type .llcrypt registered [ 45.549695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.558120] alg: No test for adler32 (adler32-zlib) [ 46.580053] Lustre: Lustre: Build Version: 2.17.51_74_g79190eb [ 46.929385] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 48.560165] Key type lgssc registered [ 49.346121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 186.370421] Lustre: Mounted lustre-client [ 192.736372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.788854] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.937938] Lustre: lustre-OST0000-osc-ffff945908509800: disconnect after 23s idle [ 216.960558] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing yml_node [ 222.204942] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 224.961405] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 227.877462] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 229.638711] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Apr 1 07:14:53 EDT 2026 [ 249.104194] LustreError: 7906:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff945908509800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 249.161312] Lustre: Unmounted lustre-client [ 279.836862] Key type lgssc unregistered [ 280.092901] LNet: 8353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.103364] LNetError: 8353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 280.155515] LNet: Removed LNI 192.168.204.40@tcp [ 280.825182] Key type .llcrypt unregistered [ 280.827252] Key type ._llcrypt unregistered [ 296.216180] hrtimer: interrupt took 5323912 ns [ 311.768441] Key type ._llcrypt registered [ 311.773248] Key type .llcrypt registered [ 312.058759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.074723] alg: No test for adler32 (adler32-zlib) [ 312.970223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 313.046808] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 313.076096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 317.013889] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 317.150310] LNet: 9225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.165464] LNetError: 9225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.188145] LNet: Removed LNI 192.168.204.40@tcp [ 318.042493] Key type .llcrypt unregistered [ 318.047417] Key type ._llcrypt unregistered [ 324.669746] Key type ._llcrypt registered [ 324.670887] Key type .llcrypt registered [ 325.840143] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:16:29 (1775042189) [ 326.313705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 326.328163] alg: No test for adler32 (adler32-zlib) [ 327.232127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.352604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.380059] LNet: 9644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.411680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.481314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.526796] LNet: 9651:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 335.590389] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:16:39 (1775042199) [ 335.745727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.777236] LNet: 10227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.835753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.944289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 343.445578] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:16:47 (1775042207) [ 343.572728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.607079] LNet: 10808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.654809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.736398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 351.369272] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:16:55 (1775042215) [ 351.679530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.720329] LNet: 11402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.753288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.855399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 359.397675] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:17:03 (1775042223) [ 359.869181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.902800] LNet: 11988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.947674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.088889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 360.697724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 368.510779] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:17:12 (1775042232) [ 368.621823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.645803] LNet: 12571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.675766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.762185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 369.408995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 369.930931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 377.094796] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:17:20 (1775042240) [ 377.534490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.596275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.714671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 384.611472] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:17:28 (1775042248) [ 384.772588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.819460] LNet: 13754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.822577] LNet: 13754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 384.861746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.926552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 384.963402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.632862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 385.656940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 386.049648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 386.105848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 386.480274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 386.517545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 386.943564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 386.969430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 387.240846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 387.282840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.628958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 387.681374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 388.018367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 388.074591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 388.510056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 388.559464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 388.942430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 388.972782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 389.291502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 389.327429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 389.557574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 389.580860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.834523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.856538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 396.131258] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:17:40 (1775042260) [ 396.397342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.447847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.564683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 396.601894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 403.445478] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:17:47 (1775042267) [ 403.739733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.808122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.899692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 403.939765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 410.865380] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:17:54 (1775042274) [ 411.021121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.101163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.264331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 411.303360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 419.202838] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:18:02 (1775042282) [ 419.316049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.361847] LNet: 16197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.376473] LNet: 16197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 419.414341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.533385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 419.580485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 426.655224] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:18:10 (1775042290) [ 426.901765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.955581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.051498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 427.102074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 433.398064] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:18:17 (1775042297) [ 433.523877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.583889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.677973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 433.734334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 440.073398] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:18:24 (1775042304) [ 440.193291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.235644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.304128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.642826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.668853] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:18:30 (1775042310) [ 446.938186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.978083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.275307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.589809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.777942] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:18:37 (1775042317) [ 453.933276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.007992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.218423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.537415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.169588] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:18:45 (1775042325) [ 461.358479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.433516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.030464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.457683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.893383] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:18:53 (1775042333) [ 470.005497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.052879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.156187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.211864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.391505] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:19:00 (1775042340) [ 476.674382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.728522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.802503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.877994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.785415] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:19:07 (1775042347) [ 483.872158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.904231] LNet: 21639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.909505] LNet: 21639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 483.941072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.018362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 489.959255] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:19:13 (1775042353) [ 490.135817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.226135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.297156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 496.566870] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:19:20 (1775042360) [ 496.664954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.720645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.813105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 497.163618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 502.956782] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:19:27 (1775042367) [ 503.094887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.159848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.241330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 503.291027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 509.574407] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:19:33 (1775042373) [ 509.767244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.810243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.939034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 509.976774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 516.810330] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:19:40 (1775042380) [ 516.923099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.978918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.100975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 517.140557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 523.226102] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:19:47 (1775042387) [ 523.387901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.455291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.540799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 523.597209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 529.702373] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:19:53 (1775042393) [ 529.996291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.047252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.136267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 530.202231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 530.217434] LNetError: 25764:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 530.257819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 530.289889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 535.740725] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:19:59 (1775042399) [ 536.023155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.060448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.117362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmsgmXO2bu [ 536.228428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmsgmXO2bu [ 536.305573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lmsgmXO2bu [ 541.891203] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:20:05 (1775042405) [ 542.139693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.196714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.263730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 542.299594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 542.348883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 548.414590] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:20:12 (1775042412) [ 548.525963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.616696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.687480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.775879] LNet: Message forwarding will be enabled in 2s [ 553.824141] LNet: Message forwarding enabled [ 554.183102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 554.208270] LNet: Message forwarding will stop in 2s to 6s [ 558.692475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 559.166936] LNet: Message forwarding disabled [ 560.710056] LNet: Message forwarding will be enabled in 2s [ 564.321549] LNet: Message forwarding enabled [ 567.025296] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:20:30 (1775042430) [ 567.171640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.519048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.584743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 574.102019] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:20:38 (1775042438) [ 574.356281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.402993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.455034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 574.492152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 574.541963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 574.605609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 574.660546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 574.707265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 574.746526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 574.778827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 574.818727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 574.865704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 574.940095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 574.986178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 575.033836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 575.071587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 575.115737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 575.150545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 575.189934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 575.243578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 575.292356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 575.321896] LNetError: 28826:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 580.746401] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:20:44 (1775042444) [ 580.932821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.992661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.104916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 586.963788] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:20:51 (1775042451) [ 588.578758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg440-server [ 590.332604] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:20:54 (1775042454) [ 590.421983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.481891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.546969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.600850] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 592.050618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[41-45]@tcp [ 592.076972] LNet: 30541:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 592.143676] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.43@tcp is being used as a gateway but routing feature is not turned on [ 592.167492] LNetError: 30329:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.43@tcp has gone from up to down [ 593.132642] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.42@tcp is being used as a gateway but routing feature is not turned on [ 593.147988] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 594.154134] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 594.170459] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 595.168809] LNetError: 30336:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.41@tcp has gone from up to down [ 595.182203] LNetError: 30336:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 596.263056] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.43@tcp is being used as a gateway but routing feature is not turned on [ 596.274356] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 598.216787] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:21:01 (1775042461) [ 600.110721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg440-server [ 600.357347] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 600.382648] LNetError: 30329:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 601.675842] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:21:05 (1775042465) [ 601.971251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.998885] LNetError: 31476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.033593] LNet: Removed LNI 192.168.204.40@tcp [ 602.071314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.120083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 602.139443] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 602.749399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 602.771677] LNet: 31593:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 602.789595] LNet: 31593:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 602.816240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 602.839857] LNetError: 31480:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.43@tcp has gone from up to down [ 609.358190] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:21:13 (1775042473) [ 609.476238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.508483] LNetError: 32181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.533601] LNet: Removed LNI 192.168.204.40@tcp [ 609.592519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.676297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 609.738983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.787819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.867292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.934180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.980378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.066395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 610.153084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.196317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.254975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.349629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.395911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.462860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 610.546524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.573822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.625288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.699059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.754856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.834979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 610.926442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.959396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.024330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.092937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.147408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.255122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 611.342716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.385563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.466894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.554754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.597884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.686971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 611.741485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.768352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.843111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.351581] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:21:22 (1775042482) [ 619.940141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg440-server [ 621.625047] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:21:25 (1775042485) [ 623.770493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg440-server [ 625.953446] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:21:29 (1775042489) [ 626.351518] LNet: 33586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.364842] LNet: 33586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 27 previous similar messages [ 627.323753] Key type .llcrypt unregistered [ 627.327758] Key type ._llcrypt unregistered [ 634.589756] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 634.854236] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 634.860215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 634.862797] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 637.543910] Key type ._llcrypt registered [ 637.547843] Key type .llcrypt registered [ 637.733649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.763911] alg: No test for adler32 (adler32-zlib) [ 638.673719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.710656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 638.725712] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 639.539883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 639.583811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.228702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 642.251743] LNet: Removed LNI 10.1.2.3@tcp [ 642.413681] LNet: 34306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.419946] LNetError: 34306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.433345] LNet: Removed LNI 192.168.204.40@tcp [ 643.067868] Key type .llcrypt unregistered [ 643.070486] Key type ._llcrypt unregistered [ 653.546492] Key type ._llcrypt registered [ 653.548389] Key type .llcrypt registered [ 654.917463] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:21:58 (1775042518) [ 655.473540] Key type .llcrypt unregistered [ 655.481993] Key type ._llcrypt unregistered [ 660.728065] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.876329] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.887350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.898525] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.521433] Key type ._llcrypt registered [ 663.523982] Key type .llcrypt registered [ 663.794588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.831497] alg: No test for adler32 (adler32-zlib) [ 664.804250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.874190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.900741] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 665.574733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 665.591400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.865316] LNet: 35799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.870558] LNetError: 35799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.874343] LNet: Removed LNI 192.168.204.40@tcp [ 667.883697] LNet: Removed LNI 10.1.2.3@tcp [ 668.520196] Key type .llcrypt unregistered [ 668.522726] Key type ._llcrypt unregistered [ 678.948877] Key type ._llcrypt registered [ 678.950816] Key type .llcrypt registered [ 680.259052] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:22:23 (1775042543) [ 680.740753] Key type .llcrypt unregistered [ 680.745220] Key type ._llcrypt unregistered [ 686.257870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 686.403805] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 686.413622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 686.423535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 688.905329] Key type ._llcrypt registered [ 688.907934] Key type .llcrypt registered [ 689.033944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.050245] alg: No test for adler32 (adler32-zlib) [ 689.987860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.045882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 690.067537] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 691.808569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 691.846863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.029352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 694.060883] LNet: Removed LNI 10.1.2.3@tcp [ 694.076433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 694.110770] LNet: Removed LNI 10.1.2.31@tcp [ 694.125863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 694.146040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.722776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 694.743796] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 695.453944] LNet: 37719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.458787] LNetError: 37719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 695.463306] LNet: Removed LNI 10.1.2.3@tcp [ 695.476106] LNet: Removed LNI 10.1.2.31@tcp [ 696.100581] Key type .llcrypt unregistered [ 696.104058] Key type ._llcrypt unregistered [ 707.390813] Key type ._llcrypt registered [ 707.397217] Key type .llcrypt registered [ 708.549091] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:22:52 (1775042572) [ 709.077437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.102830] alg: No test for adler32 (adler32-zlib) [ 710.014611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.068351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.087074] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 710.816565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.837940] LNet: 38672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.842406] LNetError: 38672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.861563] LNet: Removed LNI 192.168.204.40@tcp [ 710.872892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.906719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.40@tcp [ 710.923959] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 711.636524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.40@tcp [ 711.667738] LNet: Removed LNI 192.168.204.40@tcp [ 711.729924] LNet: 38807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.501439] Key type .llcrypt unregistered [ 712.506403] Key type ._llcrypt unregistered [ 723.342971] Key type ._llcrypt registered [ 723.346254] Key type .llcrypt registered [ 724.665918] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:23:08 (1775042588) [ 726.579598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg440-server [ 728.503457] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:23:12 (1775042592) [ 729.020897] Key type .llcrypt unregistered [ 729.023988] Key type ._llcrypt unregistered [ 734.174381] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.354918] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.364084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.371463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 736.960287] Key type ._llcrypt registered [ 736.964912] Key type .llcrypt registered [ 737.230819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.257029] alg: No test for adler32 (adler32-zlib) [ 738.169874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.234070] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 739.063870] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 741.397988] LNet: 40644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.403238] LNetError: 40644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.409427] LNet: Removed LNI 192.168.204.40@tcp1 [ 741.425151] LNet: Removed LNI 10.1.2.3@tcp2 [ 742.182785] Key type .llcrypt unregistered [ 742.187491] Key type ._llcrypt unregistered [ 753.426783] Key type ._llcrypt registered [ 753.430141] Key type .llcrypt registered [ 754.761444] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:23:38 (1775042618) [ 755.239610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 755.255561] alg: No test for adler32 (adler32-zlib) [ 756.190642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.230411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.249877] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 756.960942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.985919] LNet: 41614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.992873] LNetError: 41614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.021611] LNet: Removed LNI 192.168.204.40@tcp [ 757.038979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.091249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 757.119264] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 763.890448] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:23:47 (1775042627) [ 764.051437] LNet: 42202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.063235] LNetError: 42202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.095316] LNet: Removed LNI 192.168.204.40@tcp [ 764.678509] Key type .llcrypt unregistered [ 764.683684] Key type ._llcrypt unregistered [ 769.676918] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.846474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.852649] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.856200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.308920] Key type ._llcrypt registered [ 772.312714] Key type .llcrypt registered [ 772.481294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.497320] alg: No test for adler32 (adler32-zlib) [ 773.428572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.495140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.518900] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 774.136302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.164476] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 776.036097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.071708] LNet: 42946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.078808] LNetError: 42946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.085372] LNet: Removed LNI 192.168.204.40@tcp [ 776.098349] LNet: Removed LNI 10.1.2.3@tcp [ 776.126555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.206254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 776.231479] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 776.237276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 776.672486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.704942] LNet: 42973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.718139] LNetError: 42973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.725631] LNetError: 42973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 776.739150] LNet: Removed LNI 192.168.204.40@tcp [ 776.760779] LNet: Removed LNI 10.1.2.3@tcp [ 776.785777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.848842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 776.883591] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 776.887934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 777.333282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 777.359525] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 778.228778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.248803] LNet: 43135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.259603] LNetError: 43135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.265829] LNetError: 43135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 778.271968] LNet: Removed LNI 192.168.204.40@tcp [ 778.274863] LNet: Removed LNI 10.1.2.3@tcp [ 778.284687] LNet: Removed LNI 192.168.204.40@tcp2 [ 778.315134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.392899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 778.422740] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 778.428057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.433283] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 784.782511] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:24:08 (1775042648) [ 785.077466] LNet: 43732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.091303] LNetError: 43732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.106182] LNetError: 43732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 785.130302] LNet: Removed LNI 192.168.204.40@tcp [ 785.135137] LNet: Removed LNI 10.1.2.3@tcp [ 785.157657] LNet: Removed LNI 192.168.204.40@tcp2 [ 785.817901] Key type .llcrypt unregistered [ 785.822671] Key type ._llcrypt unregistered [ 791.382534] Key type ._llcrypt registered [ 791.386437] Key type .llcrypt registered [ 791.552471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 791.571793] alg: No test for adler32 (adler32-zlib) [ 792.482986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.534148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.556940] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 793.329322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.363258] LNet: 44126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.369372] LNetError: 44126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.385440] LNet: Removed LNI 192.168.204.40@tcp [ 793.404567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.465657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 793.502312] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 794.010280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.034912] LNet: 44156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.046628] LNetError: 44156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.068783] LNet: Removed LNI 192.168.204.40@tcp [ 794.083181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.140261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 794.179601] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 794.649721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.706510] LNet: Removed LNI 192.168.204.40@tcp [ 794.733153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.785780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 794.806486] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 795.207885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.343253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.346647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.350738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 797.771413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.797820] LNet: 44265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.804408] LNet: 44265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 797.811425] LNetError: 44265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.818197] LNetError: 44265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 797.839519] LNet: Removed LNI 192.168.204.40@tcp [ 797.855885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.948637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 797.983691] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 804.013136] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:24:28 (1775042668) [ 804.244403] LNet: 44862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.250265] LNetError: 44862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.268960] LNet: Removed LNI 192.168.204.40@tcp [ 804.908027] Key type .llcrypt unregistered [ 804.911505] Key type ._llcrypt unregistered [ 810.136344] Key type ._llcrypt registered [ 810.139090] Key type .llcrypt registered [ 810.288448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.300537] alg: No test for adler32 (adler32-zlib) [ 811.156625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.221365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.245502] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 812.064654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.080146] LNet: 45260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.084563] LNetError: 45260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.098665] LNet: Removed LNI 192.168.204.40@tcp [ 812.117444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.205631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 812.247642] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 817.871465] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:24:41 (1775042681) [ 818.125793] LNet: 45852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.133507] LNetError: 45852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.155474] LNet: Removed LNI 192.168.204.40@tcp [ 818.640273] Key type .llcrypt unregistered [ 818.642670] Key type ._llcrypt unregistered [ 824.380469] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.501925] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.508218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.511948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 827.019621] Key type ._llcrypt registered [ 827.028431] Key type .llcrypt registered [ 827.244486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.266135] alg: No test for adler32 (adler32-zlib) [ 828.171143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.226248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.246624] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 828.941460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.961455] LNet: 46301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.966669] LNetError: 46301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.994546] LNet: Removed LNI 192.168.204.40@tcp [ 829.035454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.091832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 829.116767] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 834.961631] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:24:58 (1775042698) [ 835.043629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.062219] LNet: 46894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.070539] LNetError: 46894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.087827] LNet: Removed LNI 192.168.204.40@tcp [ 835.113684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.197463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 840.366174] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:25:04 (1775042704) [ 840.450785] LNet: 47472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.070635] Key type .llcrypt unregistered [ 841.073742] Key type ._llcrypt unregistered [ 846.146834] Key type ._llcrypt registered [ 846.150159] Key type .llcrypt registered [ 846.352646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.365884] alg: No test for adler32 (adler32-zlib) [ 847.280693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.338631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.357392] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 848.210447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.226183] LNet: 47878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.232873] LNetError: 47878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.247754] LNet: Removed LNI 192.168.204.40@tcp [ 848.273598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.331447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 848.353447] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 853.914955] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:25:17 (1775042717) [ 854.083721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.101761] LNet: 48469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.109565] LNetError: 48469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.130799] LNet: Removed LNI 192.168.204.40@tcp [ 854.157814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.230542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 859.730396] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:25:23 (1775042723) [ 861.112671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 862.699210] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:25:26 (1775042726) [ 864.077688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 865.727249] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:25:29 (1775042729) [ 866.962375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 868.775355] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:25:32 (1775042732) [ 870.202157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 871.847343] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:25:35 (1775042735) [ 873.188166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 874.935973] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:25:38 (1775042738) [ 875.040656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.082643] LNet: 50516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.115064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.234402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.257258] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 876.007478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.40 [ 876.025340] LNet: 50655:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 876.033541] LNet: Removed LNI 192.168.204.40@tcp [ 881.882553] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:25:45 (1775042745) [ 881.955877] LNet: 51218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.960846] LNetError: 51218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.992599] LNet: Removed LNI 192.168.204.40@tcp [ 882.619806] Key type .llcrypt unregistered [ 882.624688] Key type ._llcrypt unregistered [ 887.204080] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 887.419375] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 887.424989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 887.428440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.165885] Key type ._llcrypt registered [ 890.167803] Key type .llcrypt registered [ 890.332681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.362401] alg: No test for adler32 (adler32-zlib) [ 891.808553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.437794] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:26:01 (1775042761) [ 897.702387] LNet: 52131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.711272] LNetError: 52131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.735749] LNet: Removed LNI 10.1.2.3@tcp [ 898.328357] Key type .llcrypt unregistered [ 898.331940] Key type ._llcrypt unregistered [ 902.621671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 902.824802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 902.829783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 902.834410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 905.417296] Key type ._llcrypt registered [ 905.420376] Key type .llcrypt registered [ 905.565252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.579164] alg: No test for adler32 (adler32-zlib) [ 906.500509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 912.279215] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:26:16 (1775042776) [ 912.369360] LNet: 53016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.375975] LNetError: 53016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.399354] LNet: Removed LNI 10.1.2.3@tcp [ 913.064242] Key type .llcrypt unregistered [ 913.066910] Key type ._llcrypt unregistered [ 918.045893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.248733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.255586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.260813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.941187] Key type ._llcrypt registered [ 920.943096] Key type .llcrypt registered [ 921.127107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.157317] alg: No test for adler32 (adler32-zlib) [ 922.121388] LNet: 53330:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 922.141424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.537746] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:26:31 (1775042791) [ 927.746641] LNet: 53922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.753688] LNetError: 53922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.768984] LNet: Removed LNI 10.1.2.3@tcp [ 928.315550] Key type .llcrypt unregistered [ 928.317370] Key type ._llcrypt unregistered [ 932.678925] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.889939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.897360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.902466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 935.478810] Key type ._llcrypt registered [ 935.482558] Key type .llcrypt registered [ 935.620035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.630352] alg: No test for adler32 (adler32-zlib) [ 936.534576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.582113] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 943.449463] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:26:47 (1775042807) [ 943.905182] LNet: 55151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.913853] LNetError: 55151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 943.943721] LNet: Removed LNI 10.1.2.3@tcp [ 944.460612] Key type .llcrypt unregistered [ 944.463830] Key type ._llcrypt unregistered [ 949.018958] Key type ._llcrypt registered [ 949.022289] Key type .llcrypt registered [ 949.185850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.197882] alg: No test for adler32 (adler32-zlib) [ 950.096264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 950.119732] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 956.519389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 957.227858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 957.280469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 957.322751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 957.345689] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 957.587382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.623483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 957.648846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.693431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.722390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.767442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.809296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.857474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.885711] LNet: 55588:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.901977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.926322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.948606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.975224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.007293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.040877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.068344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 958.516684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 958.563325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 958.600832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 958.623985] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 958.834566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.859878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 958.890628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.920359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.952603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.992016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.026293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.058313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 959.076359] LNet: 55697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 959.093773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.121356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.150619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.176877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.207703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.233726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.258294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 959.662225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 959.697438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 959.725952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 959.739948] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 959.909496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 959.934200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 959.961432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 959.984967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 960.005996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 960.031543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 960.057610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 960.082472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.097717] LNet: 55807:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 960.112397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.137590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.161631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.187991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.219543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.251245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.287507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 960.679950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 960.710670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 960.737135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 960.953960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 960.977422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 961.011181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.044350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.074609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.116133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.155340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.193994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.231176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.284927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.321433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.354622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.381173] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 961.389983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 961.409809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 961.432415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 961.889573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 961.925025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 961.972750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 962.003749] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 962.012876] LNet: Skipped 2 previous similar messages [ 962.211291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.245562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 962.285568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 962.317277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 962.345787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 962.386646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.420450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.472639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 962.501531] LNet: 56025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 962.509988] LNet: 56025:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 962.525991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 962.553489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 962.605044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 962.642929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 962.688912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 962.737507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 962.774729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 963.188622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 963.233815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 963.268637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 963.446520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 963.476921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 963.510066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 963.548703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 963.581784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 963.612792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 963.650275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 963.698460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.728738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.761118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.792719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.823642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.861767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.906725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.934260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 964.142530] LNet: 56175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.149961] LNetError: 56175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.226471] LNet: Removed LNI 192.168.204.40@tcp [ 965.598728] Key type .llcrypt unregistered [ 965.602212] Key type ._llcrypt unregistered [ 977.776399] Key type ._llcrypt registered [ 977.778203] Key type .llcrypt registered [ 978.846061] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:27:22 (1775042842) [ 979.582298] Key type .llcrypt unregistered [ 979.584108] Key type ._llcrypt unregistered [ 983.978645] Key type ._llcrypt registered [ 983.981917] Key type .llcrypt registered [ 984.152852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.168655] alg: No test for adler32 (adler32-zlib) [ 985.062744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 985.088049] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 991.058990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 999.536962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.587438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 999.620262] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1000.870206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1000.916250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1000.949583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1000.980933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1001.024536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1001.067085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1001.111142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1001.146642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1001.193544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1001.224401] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1001.234482] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1001.442893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1001.467007] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1001.498842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1001.527126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1001.579370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1001.608118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1001.641031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1001.682328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1001.699667] LNet: 57639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1001.714882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1001.743411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1001.767872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1001.799489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1001.828886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1001.856165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1001.878470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1002.302794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1002.339562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1002.375578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1002.409053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1002.440513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1002.475172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1002.509971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1002.549671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1002.578913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1002.607121] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1002.609996] LNet: Skipped 2 previous similar messages [ 1002.780658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.808773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1002.843920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.889913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.922475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.957645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.984532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.007584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.019544] LNet: 57761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.036563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.058151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.079600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.099155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.125290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.161058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.215502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1003.745724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1003.772278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1003.800738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1003.826757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1003.852709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1003.882347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1003.912680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1003.941335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1003.969548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1003.982882] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1003.987313] LNet: Skipped 2 previous similar messages [ 1004.205259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.230865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1004.262679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1004.284311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.314646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.339199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.383934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1004.426664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1004.453750] LNet: 57882:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1004.484524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1004.532718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1004.566480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1004.612640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1004.649728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.690297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.721792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1005.167640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1005.195766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1005.228735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1005.252815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1005.277325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1005.314289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1005.352387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1005.398558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1005.436952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1005.682778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1005.713739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1005.747799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.771765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.799132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1005.833090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.875195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.900593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.957058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.997161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.034137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.064882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.096766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.148861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.187871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1006.651366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1006.700487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1006.768712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1006.810566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1006.853871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1006.906701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1006.950444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1006.989599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1007.023679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1007.044952] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1007.050909] LNet: Skipped 5 previous similar messages [ 1007.235747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1007.255880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1007.276638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.300989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.322962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.345135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1007.369803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1007.391423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1007.405764] LNet: 58124:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1007.410917] LNet: 58124:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1007.422423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.463597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.501324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.531136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.564512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.601576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.641897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1008.019753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1008.060534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1008.093654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1008.129297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1008.168534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1008.217675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1008.263653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1008.305373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1008.336461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1008.552606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1008.590901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1008.619904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.649754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.685052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1008.735305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.784511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.812735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.837236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.868556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.901903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.932570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.959527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.988357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1009.021713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.175893] LNet: 58321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.181497] LNetError: 58321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.186840] LNet: Removed LNI 192.168.204.40@tcp [ 1013.196774] LNet: Removed LNI 192.168.204.40@tcp1 [ 1013.504578] Key type .llcrypt unregistered [ 1013.509353] Key type ._llcrypt unregistered [ 1025.987753] Key type ._llcrypt registered [ 1025.993209] Key type .llcrypt registered [ 1026.967910] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:28:11 (1775042891) [ 1027.762590] Key type .llcrypt unregistered [ 1027.766747] Key type ._llcrypt unregistered [ 1032.673723] Key type ._llcrypt registered [ 1032.677251] Key type .llcrypt registered [ 1032.904385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.929328] alg: No test for adler32 (adler32-zlib) [ 1033.814621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1033.835441] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1039.894607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1040.422889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1040.466882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1040.529075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1040.557377] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1040.743200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.770980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1040.795393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.836164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.868993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.896147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.926878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.957695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.977179] LNet: 59569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.991820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.027877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.055285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.088575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.145978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.180685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.214633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1041.648937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1041.689358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1041.730329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1041.749505] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1041.950623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.983426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.010331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.061508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.089390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.126916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.168800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.194378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.204705] LNet: 59678:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.216269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.235839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.265909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.291566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.322207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.365610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.414634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1042.848366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1042.889586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1042.936186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1042.960917] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1043.148757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.186480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.214479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.240938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.275547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.301921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.337348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.365438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.388502] LNet: 59787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.404644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.431464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.456871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.479803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.504533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.530511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.559372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.755767] LNet: 59828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.760633] LNetError: 59828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.777854] LNet: Removed LNI 192.168.204.40@tcp [ 1045.085452] Key type .llcrypt unregistered [ 1045.087526] Key type ._llcrypt unregistered [ 1057.028273] Key type ._llcrypt registered [ 1057.031449] Key type .llcrypt registered [ 1058.070842] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:28:42 (1775042922) [ 1058.758321] Key type .llcrypt unregistered [ 1058.761598] Key type ._llcrypt unregistered [ 1063.176610] Key type ._llcrypt registered [ 1063.182096] Key type .llcrypt registered [ 1063.331217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.346685] alg: No test for adler32 (adler32-zlib) [ 1064.247608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1064.264936] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1070.308725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1077.267444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.310761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1077.335408] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1078.566195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1078.593882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1078.620316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1078.670220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1078.698344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1078.732632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1078.787974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1078.835387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1078.876735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1078.910728] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1078.916746] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1079.077849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.105904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1079.133338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.161992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.188790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.228631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.254986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.288386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.308021] LNet: 61295:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.323259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.351425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.397437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.425731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.453521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.481969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.514703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.548687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.901876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1079.936871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1079.972929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1080.004787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1080.038383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1080.078076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1080.105346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1080.132231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1080.172288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1080.208360] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1080.214744] LNet: Skipped 2 previous similar messages [ 1080.390574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.417313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1080.444742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.470730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.503389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.529358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.550364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.580937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.595656] LNet: 61418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.608520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.632654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.663284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.687850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.712544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.742776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.777750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1080.796497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.176681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1081.209631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1081.242552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1081.303423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1081.352737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1081.387592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1081.427421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1081.463627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1081.493464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1081.509114] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1081.635388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.663837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1081.686922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.727587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.761573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.798372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.829564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.849649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.866445] LNet: 61541:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.879689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.916776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.963622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.997590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.033606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.066591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.096332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.121996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.639405] LNet: 61619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.644757] LNetError: 61619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.651177] LNet: Removed LNI 192.168.204.40@tcp [ 1085.659632] LNet: Removed LNI 192.168.204.40@tcp1 [ 1085.930877] Key type .llcrypt unregistered [ 1085.936058] Key type ._llcrypt unregistered [ 1095.706161] Key type ._llcrypt registered [ 1095.708773] Key type .llcrypt registered [ 1096.560884] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:29:20 (1775042960) [ 1096.991369] Key type .llcrypt unregistered [ 1096.993444] Key type ._llcrypt unregistered [ 1100.831414] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1100.977490] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1100.981804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1100.985635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1103.358763] Key type ._llcrypt registered [ 1103.360898] Key type .llcrypt registered [ 1103.478607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1103.487326] alg: No test for adler32 (adler32-zlib) [ 1104.349757] LNet: 62695:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1104.364726] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1104.394746] LNet: 62709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.400556] LNetError: 62709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.412130] LNet: Removed LNI 192.168.204.40@tcp [ 1104.860473] Key type .llcrypt unregistered [ 1104.864426] Key type ._llcrypt unregistered [ 1108.441986] Key type ._llcrypt registered [ 1108.443820] Key type .llcrypt registered [ 1108.547949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.558737] alg: No test for adler32 (adler32-zlib) [ 1109.407850] LNet: 62968:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1109.421266] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1109.424316] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1109.445772] LNet: 62982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.450328] LNetError: 62982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.453597] LNet: Removed LNI 192.168.204.40@tcp [ 1109.463389] LNet: Removed LNI 10.1.2.3@tcp1 [ 1109.903416] Key type .llcrypt unregistered [ 1109.907570] Key type ._llcrypt unregistered [ 1113.292935] Key type ._llcrypt registered [ 1113.294726] Key type .llcrypt registered [ 1113.407512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.422249] alg: No test for adler32 (adler32-zlib) [ 1114.278335] LNet: 63241:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1114.290535] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1114.293463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1114.325443] LNet: 63255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.332619] LNetError: 63255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.337801] LNet: Removed LNI 192.168.204.40@tcp [ 1114.347139] LNet: Removed LNI 10.1.2.3@tcp [ 1114.795643] Key type .llcrypt unregistered [ 1114.797181] Key type ._llcrypt unregistered [ 1118.549312] Key type ._llcrypt registered [ 1118.553440] Key type .llcrypt registered [ 1118.700957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.712418] alg: No test for adler32 (adler32-zlib) [ 1119.567391] LNet: 63514:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1119.576057] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1119.579179] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.602855] LNet: 63528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.607834] LNetError: 63528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.612620] LNet: Removed LNI 192.168.204.40@tcp [ 1119.622772] LNet: Removed LNI 10.1.2.3@tcp [ 1120.106721] Key type .llcrypt unregistered [ 1120.108925] Key type ._llcrypt unregistered [ 1124.087940] Key type ._llcrypt registered [ 1124.091222] Key type .llcrypt registered [ 1124.210801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.221560] alg: No test for adler32 (adler32-zlib) [ 1125.111528] LNet: 63792:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1125.124376] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1125.128158] LNetError: 63792:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1125.133120] LNetError: Error -100 starting up LNI tcp [ 1125.144895] LNet: Removed LNI 192.168.204.40@tcp [ 1125.594368] Key type .llcrypt unregistered [ 1125.596087] Key type ._llcrypt unregistered [ 1134.142846] Key type ._llcrypt registered [ 1134.149908] Key type .llcrypt registered [ 1135.227599] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:29:59 (1775042999) [ 1135.862378] Key type .llcrypt unregistered [ 1135.867071] Key type ._llcrypt unregistered [ 1140.005759] Key type ._llcrypt registered [ 1140.008104] Key type .llcrypt registered [ 1140.164348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.177310] alg: No test for adler32 (adler32-zlib) [ 1141.076753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1141.092952] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1147.475964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1148.202195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1148.233164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1148.266289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1148.291927] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1148.507685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.532813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1148.558918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.593561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.633804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.664497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.692433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.736905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.758436] LNet: 65048:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.776605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.805233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.849987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.879847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.910467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.953948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.178725] LNet: 65087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.185876] LNetError: 65087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.202577] LNet: Removed LNI 192.168.204.40@tcp [ 1149.503621] Key type .llcrypt unregistered [ 1149.505492] Key type ._llcrypt unregistered [ 1163.809230] Key type ._llcrypt registered [ 1163.810942] Key type .llcrypt registered [ 1163.986986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.998277] alg: No test for adler32 (adler32-zlib) [ 1164.890539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1164.914261] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1171.310352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1178.075455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.116511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1178.137508] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1179.187707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1179.226506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1179.264947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1179.311827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1179.336767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1179.383170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1179.423505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1179.455382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1179.486173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1179.508548] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1179.675765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.708818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1179.749423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.796914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.824896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.848516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.883036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.910729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.929200] LNet: 65965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.947618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.971699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.994715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.017376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.065723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.098462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.583905] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1183.884920] LNet: 66038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.898674] LNetError: 66038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.928194] LNet: Removed LNI 192.168.204.40@tcp [ 1184.932976] LNetError: 66038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.951316] LNet: Removed LNI 192.168.204.40@tcp1 [ 1185.252487] Key type .llcrypt unregistered [ 1185.255497] Key type ._llcrypt unregistered [ 1197.384211] Key type ._llcrypt registered [ 1197.385928] Key type .llcrypt registered [ 1198.284904] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:31:02 (1775043062) [ 1198.652946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.669857] alg: No test for adler32 (adler32-zlib) [ 1199.574571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.616755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1199.640408] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1200.192742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1200.210849] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1200.828430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1200.892889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1200.936024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1200.983386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1201.017988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1201.053495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1201.082337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1201.109799] LNetError: 67137:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1201.633725] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1201.635144] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1203.681911] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1204.509748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1207.776457] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1208.949746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1208.989633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1209.046644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.073097] LNet: 67255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.080434] LNetError: 67255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.144298] LNet: Removed LNI 192.168.204.40@tcp [ 1210.146719] LNetError: 67255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.161345] LNet: Removed LNI 192.168.204.40@tcp1 [ 1210.186769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.241478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1210.260934] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1210.816463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1210.830333] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1211.493228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1211.520715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.577526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1211.641415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1211.669536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1211.703309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1211.736832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1211.771058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1211.786401] LNetError: 67527:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1212.256705] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1215.139138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1218.403781] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1218.422645] LNet: Skipped 1 previous similar message [ 1222.874465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1222.902583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1222.930314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.357618] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:31:32 (1775043092) [ 1228.565787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.591045] LNet: 68244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.595333] LNetError: 68244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.601866] LNet: Removed LNI 192.168.204.40@tcp [ 1228.611795] LNet: Removed LNI 192.168.204.40@tcp1 [ 1228.627617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.678987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1228.701176] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1229.283880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1229.298100] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1229.990770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1230.051434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1230.080631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1230.114174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1230.147594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1230.177695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1230.212388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1230.312727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1230.688269] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.40@tcp [ 1231.712365] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1231.716906] LNet: Skipped 1 previous similar message [ 1245.274870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1245.399851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.422095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.443446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 500 [ 1245.522242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1252.065556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1252.151254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1252.193986] LNet: 68735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.198722] LNetError: 68735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.207021] LNetError: 68735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1252.221210] LNet: Removed LNI 192.168.204.40@tcp [ 1252.241457] LNet: Removed LNI 192.168.204.40@tcp1 [ 1252.257150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.332214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1252.372201] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1253.000860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.023529] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1253.717556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1253.746875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.787100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1253.814024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1253.855337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1253.886728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1253.909685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1253.947160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1254.372536] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1254.380039] LNet: Skipped 3 previous similar messages [ 1255.197607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.278539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.330389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.376374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.951765] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:32:15 (1775043135) [ 1271.325823] LNet: 69780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.335992] LNetError: 69780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.344046] LNetError: 69780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1271.353562] LNet: Removed LNI 192.168.204.40@tcp [ 1271.366302] LNet: Removed LNI 192.168.204.40@tcp1 [ 1271.905893] Key type .llcrypt unregistered [ 1271.908956] Key type ._llcrypt unregistered [ 1276.725174] Key type ._llcrypt registered [ 1276.729664] Key type .llcrypt registered [ 1276.862994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.876496] alg: No test for adler32 (adler32-zlib) [ 1277.783714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.811477] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1284.392458] Lustre: DEBUG MARKER: Initial discovery [ 1284.422273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1285.870111] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1286.009540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1286.036596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1286.068257] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1286.084954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1286.123845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -l 3 [ 1286.161550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -e local_error [ 1286.203085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m PUT -l 6 [ 1287.075681] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1287.241292] Lustre: DEBUG MARKER: Force oleg440-client.virtnet to discover 192.168.204.140@tcp (in background) [ 1287.448336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1288.614708] Lustre: DEBUG MARKER: Force oleg440-server to discover 192.168.204.40@tcp [ 1290.315628] Lustre: DEBUG MARKER: Wait for 70542 [ 1290.531892] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1291.488953] Lustre: DEBUG MARKER: Finished wait on 70542 [ 1291.572570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1291.604078] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1291.608558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1291.791410] LNet: 70991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.797832] LNetError: 70991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.810695] LNet: Removed LNI 192.168.204.40@tcp [ 1292.025805] Key type .llcrypt unregistered [ 1292.027788] Key type ._llcrypt unregistered [ 1304.160730] Key type ._llcrypt registered [ 1304.162779] Key type .llcrypt registered [ 1305.064261] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:32:49 (1775043169) [ 1305.605505] Key type .llcrypt unregistered [ 1305.609044] Key type ._llcrypt unregistered [ 1310.168639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1310.337922] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1310.343656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1310.349199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1312.725394] Key type ._llcrypt registered [ 1312.728043] Key type .llcrypt registered [ 1312.866703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.881309] alg: No test for adler32 (adler32-zlib) [ 1313.765024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.805515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1313.822058] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1314.367239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1314.382318] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1321.165521] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:33:05 (1775043185) [ 1321.338424] LNet: 73066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.346217] LNetError: 73066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.353123] LNet: Removed LNI 192.168.204.40@tcp [ 1321.360627] LNet: Removed LNI 10.1.2.3@tcp [ 1322.056432] Key type .llcrypt unregistered [ 1322.060277] Key type ._llcrypt unregistered [ 1327.945089] Key type ._llcrypt registered [ 1327.948300] Key type .llcrypt registered [ 1328.145942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.155340] alg: No test for adler32 (adler32-zlib) [ 1329.027904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.087648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.111507] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1329.681350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1329.701464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1330.532659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.40@tcp [ 1331.620172] LNetError: Unexpected error -22 connecting to 192.168.204.40@tcp at host 192.168.204.40:988 [ 1335.391883] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:33:19 (1775043199) [ 1335.579564] LNet: 74197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.584788] LNetError: 74197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.589781] LNet: Removed LNI 192.168.204.40@tcp [ 1335.602567] LNet: Removed LNI 10.1.2.3@tcp [ 1336.001743] Key type .llcrypt unregistered [ 1336.005124] Key type ._llcrypt unregistered [ 1340.113808] Key type ._llcrypt registered [ 1340.115717] Key type .llcrypt registered [ 1340.256523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.286721] alg: No test for adler32 (adler32-zlib) [ 1341.150575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.184697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1341.203504] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1341.711293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1341.742652] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1342.369339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.40@tcp1 --nid 192.168.204.40@tcp2 [ 1345.099343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp2 [ 1345.117387] LNetError: 75031:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.40@tcp1 to 192.168.204.40@tcp2 - Route table may be misconfigured [ 1345.135183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp2 192.168.204.40@tcp1 [ 1349.905387] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:33:33 (1775043213) [ 1350.154618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1350.174891] LNet: 75604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.182861] LNetError: 75604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.200256] LNet: Removed LNI 192.168.204.40@tcp1 [ 1351.202312] LNetError: 75604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.230503] LNet: Removed LNI 192.168.204.40@tcp2 [ 1351.269309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.315928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.348712] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1351.865873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.887896] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1352.495279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1352.521176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.40@tcp [ 1352.550596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp1 [ 1352.593485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp -e network_timeout [ 1352.620462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -e network_timeout [ 1352.642663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -e network_timeout [ 1352.663226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -e network_timeout [ 1352.687118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1352.702427] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1352.715210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1352.781784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1352.827327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.378445] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1357.817963] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:33:41 (1775043221) [ 1358.097926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.117928] LNet: 76465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.124188] LNetError: 76465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.138947] LNet: Removed LNI 192.168.204.40@tcp [ 1360.167179] LNet: Removed LNI 192.168.204.40@tcp1 [ 1360.190156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.253359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1360.424776] LNet: 76503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.711947] Key type .llcrypt unregistered [ 1360.719219] Key type ._llcrypt unregistered [ 1369.485738] Key type ._llcrypt registered [ 1369.488676] Key type .llcrypt registered [ 1370.591445] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:33:54 (1775043234) [ 1370.865147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.879171] alg: No test for adler32 (adler32-zlib) [ 1371.764394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.999315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1374.185733] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:33:58 (1775043238) [ 1374.333742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.368166] LNet: 77600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.395220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.443575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.472845] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1375.048083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.063358] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1375.660799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1375.686195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp1 [ 1375.714221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp1 [ 1380.078215] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:34:04 (1775043244) [ 1381.634262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg440-server [ 1382.849994] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:34:07 (1775043247) [ 1384.301324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg440-server [ 1385.623823] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:34:09 (1775043249) [ 1387.132818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg440-server [ 1388.503359] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:34:12 (1775043252) [ 1390.008993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg440-server [ 1391.391708] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:34:15 (1775043255) [ 1392.651348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg440-server [ 1394.060854] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:34:18 (1775043258) [ 1395.397110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg440-server [ 1396.663715] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:34:20 (1775043260) [ 1398.244437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg440-server [ 1399.574780] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:34:23 (1775043263) [ 1401.028980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg440-server [ 1402.269888] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:34:26 (1775043266) [ 1403.581139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg440-server [ 1404.850689] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:34:29 (1775043269) [ 1404.945657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.967869] LNet: 81472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.973726] LNetError: 81472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.982216] LNet: Removed LNI 192.168.204.40@tcp [ 1406.055193] LNet: Removed LNI 192.168.204.40@tcp1 [ 1406.085667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.142329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.186209] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1406.721362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1406.787021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1406.941988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.957555] LNet: 81624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.964825] LNetError: 81624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.972380] LNetError: 81624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1406.986174] LNet: Removed LNI 192.168.204.40@tcp [ 1407.005463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.048964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1407.068864] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1407.659772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1407.792996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.874311] LNet: Removed LNI 192.168.204.40@tcp [ 1408.920774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.980364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.016837] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1409.592577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1409.644355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1409.765967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.786431] LNet: 81916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.793656] LNet: 81916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1409.799063] LNetError: 81916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.802584] LNetError: 81916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1409.813142] LNet: Removed LNI 192.168.204.40@tcp [ 1409.838745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.886864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1409.910889] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1410.483739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1410.635834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.688065] LNet: Removed LNI 192.168.204.40@tcp [ 1411.713035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.761139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.784235] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1412.339168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1412.396324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1412.550735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.563763] LNetError: 82209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.568054] LNetError: 82209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1412.578479] LNet: Removed LNI 192.168.204.40@tcp [ 1412.599959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.634586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1412.658846] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1413.139676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1413.291240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1414.377223] LNet: Removed LNI 192.168.204.40@tcp [ 1414.399723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.459483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1414.489861] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1415.038462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1415.142089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1415.368371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.392979] LNet: 82501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.398544] LNet: 82501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1416.424932] LNet: Removed LNI 192.168.204.40@tcp [ 1416.465834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.507315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1416.526597] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1417.059770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1417.213278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.228479] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.233683] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1417.245630] LNet: Removed LNI 192.168.204.40@tcp [ 1417.262607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.308813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1417.327075] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1417.770273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1417.831524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1417.994822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.047590] LNet: Removed LNI 192.168.204.40@tcp [ 1419.066238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.112543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1419.135162] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1419.681422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1419.851847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.908306] LNet: Removed LNI 192.168.204.40@tcp [ 1420.925565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.979695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.001096] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1421.443714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1421.499644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1421.661808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.695395] LNet: Removed LNI 192.168.204.40@tcp [ 1422.718469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.765421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1422.796869] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1423.311737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1423.530308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.549783] LNet: 83231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.555904] LNet: 83231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1424.615395] LNet: Removed LNI 192.168.204.40@tcp [ 1424.634170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.680872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.704102] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1425.179515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1425.230524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.247900] LNetError: 83364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.253307] LNetError: 83364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1425.266996] LNet: Removed LNI 192.168.204.40@tcp [ 1425.282463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.327286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1425.354457] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1425.907886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1430.107926] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:34:54 (1775043294) [ 1430.220079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.246679] LNet: Removed LNI 192.168.204.40@tcp [ 1430.263241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.313525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1430.329538] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1431.092318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.119706] LNet: Removed LNI 192.168.204.40@tcp231 [ 1431.134638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.181372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1431.195329] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1431.198523] LNet: 84213:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.531702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1431.559097] LNet: Removed LNI 192.168.204.40@tcp231 [ 1431.578368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1431.605568] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1432.315363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1432.332608] LNet: Removed LNI 192.168.204.40@tcp231 [ 1432.346925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.397575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1432.413805] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1432.418700] LNet: 84366:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.743046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1432.794795] LNet: Removed LNI 192.168.204.40@tcp231 [ 1432.809344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1432.826338] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1433.648465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.670216] LNet: Removed LNI 192.168.204.40@tcp231 [ 1433.689680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.754283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1433.795782] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1433.803727] LNet: 84517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.155681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1434.185942] LNet: Removed LNI 192.168.204.40@tcp231 [ 1434.206575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1434.232991] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1434.895353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.917532] LNet: Removed LNI 192.168.204.40@tcp231 [ 1434.938932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.000390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1435.021346] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1435.329648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1435.366365] LNet: Removed LNI 192.168.204.40@tcp231 [ 1435.377647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1435.393449] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1440.196073] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:35:04 (1775043304) [ 1440.277189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.305231] LNet: 85371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.310633] LNet: 85371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1440.333739] LNet: Removed LNI 192.168.204.40@tcp231 [ 1440.354728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.430514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1440.456159] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1440.943349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1440.972854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1440.997763] LNet: Removed LNI 192.168.204.40@tcp232 [ 1441.018673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1441.042678] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1441.547109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1441.569310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1441.596327] LNet: Removed LNI 192.168.204.40@tcp232 [ 1441.608394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1441.633286] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1442.098223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1442.139451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1442.170773] LNet: Removed LNI 192.168.204.40@tcp232 [ 1442.184786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1442.209860] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1442.663748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1442.692282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1442.718539] LNet: Removed LNI 192.168.204.40@tcp232 [ 1442.732346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1446.750794] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:35:11 (1775043311) [ 1447.525506] Key type .llcrypt unregistered [ 1447.527605] Key type ._llcrypt unregistered [ 1451.833806] Key type ._llcrypt registered [ 1451.836417] Key type .llcrypt registered [ 1452.007520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.015795] alg: No test for adler32 (adler32-zlib) [ 1452.898620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1452.919982] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1458.437976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1464.398980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.432048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.454695] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1465.394097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1465.424528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1465.449582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1465.476436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1465.514471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1465.550667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1465.577689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1465.605566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1465.632578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1465.652686] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1465.660210] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1465.806853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1465.826904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1465.849971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1465.872923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1465.896145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1465.917513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1465.949643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1465.972071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1465.991151] LNet: 87167:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.010596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1466.036166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1466.057537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.086745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.117184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1466.140302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1466.461558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1466.489572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1466.519815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1466.555525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1466.596281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1466.636452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1466.662515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1466.698428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1466.742069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1466.768099] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1466.941832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1466.962876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1466.985590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1467.012199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1467.036391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1467.055425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1467.080894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1467.113134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1467.131962] LNet: 87287:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1467.147869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1467.188749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1467.227971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1467.256798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1467.289319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1467.334173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1467.721786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1467.751712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1467.774699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1467.810288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1467.837363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1467.861889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1467.907168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1467.948332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1467.990166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1468.002664] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1468.207769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1468.233209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1468.262346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1468.290203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1468.314495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1468.341933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1468.362794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1468.390379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1468.406578] LNet: 87407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.417889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1468.441867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1468.463532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.489452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1468.516302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1468.552903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1468.933508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1468.970476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1469.006244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1469.026699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1469.052066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1469.088622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1469.127854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1469.159600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1469.184572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1469.375656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1469.401504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1469.427973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1469.446263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1469.471412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1469.500171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1469.528905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1469.559409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1469.587511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1469.612116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1469.639322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.669934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1469.703969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1469.726175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1470.092847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1470.124855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1470.157591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1470.203427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1470.238590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1470.270143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1470.301489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1470.332934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1470.369391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1470.396803] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1470.400793] LNet: Skipped 1 previous similar message [ 1470.591605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1470.623365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1470.645577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1470.678147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1470.709979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1470.738222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1470.760810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1470.786876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1470.807571] LNet: 87647:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1470.816655] LNet: 87647:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1470.837575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1470.873304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1470.898909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1470.935852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1470.964922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1471.000897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1471.363686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1471.391697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1471.423906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1471.471460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1471.501847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1471.531775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1471.555953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1471.590448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1471.617673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1471.633644] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1471.795135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1471.821283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1471.854647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1471.883490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1471.922094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1471.953807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1471.987317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1472.020802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1472.058450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1472.102452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1472.144987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1472.185064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1472.238653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1472.266600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1475.821597] LNet: 87841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.827537] LNetError: 87841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.833442] LNet: Removed LNI 192.168.204.40@tcp [ 1475.843846] LNet: Removed LNI 192.168.204.40@tcp1 [ 1476.123301] Key type .llcrypt unregistered [ 1476.126506] Key type ._llcrypt unregistered [ 1487.751130] Key type ._llcrypt registered [ 1487.753092] Key type .llcrypt registered [ 1488.715444] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:35:52 (1775043352) [ 1489.035283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.048534] alg: No test for adler32 (adler32-zlib) [ 1489.913605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.964805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.993297] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1490.667837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1490.698888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1490.715078] LNetError: 88818:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1490.784162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.40@tcp [ 1490.849202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1495.505880] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:35:59 (1775043359) [ 1495.599768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.621414] LNet: 89401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.628845] LNetError: 89401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.651288] LNet: Removed LNI 192.168.204.40@tcp [ 1495.676843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.747648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.797337] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1496.535983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1496.554726] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1497.421473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.40@tcp [ 1497.446052] LNetError: 89673:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1497.760353] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1497.765491] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1498.672238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.40@tcp [ 1504.249927] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:36:08 (1775043368) [ 1505.795337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg440-server [ 1507.103716] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:36:11 (1775043371) [ 1508.496076] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg440-server [ 1509.777167] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:36:13 (1775043373) [ 1511.302895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg440-server [ 1512.604542] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:36:16 (1775043376) [ 1512.792735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.810341] LNet: 91297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.820747] LNetError: 91297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.828693] LNet: Removed LNI 192.168.204.40@tcp [ 1512.843652] LNet: Removed LNI 192.168.204.40@tcp1 [ 1512.858563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.906251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.936336] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1513.504653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1513.557456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1513.590808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1513.642312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1513.688141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1513.738580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1518.504779] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:36:22 (1775043382) [ 1518.760592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.792621] LNet: 92022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.796964] LNetError: 92022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.802736] LNetError: 92022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1518.821839] LNet: Removed LNI 192.168.204.40@tcp [ 1518.840093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.894142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.916809] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1524.928130] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:36:28 (1775043388) [ 1524.997838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.018342] LNet: 92720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.022874] LNetError: 92720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.044638] LNet: Removed LNI 192.168.204.40@tcp [ 1525.065120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.130926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.150108] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1531.321724] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:36:35 (1775043395) [ 1532.590680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1533.866944] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:36:38 (1775043398) [ 1534.440757] LNet: 93761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.448941] LNetError: 93761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.464140] LNet: Removed LNI 192.168.204.40@tcp [ 1534.943730] Key type .llcrypt unregistered [ 1534.946548] Key type ._llcrypt unregistered [ 1539.606159] Key type ._llcrypt registered [ 1539.612105] Key type .llcrypt registered [ 1539.807599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1539.820390] alg: No test for adler32 (adler32-zlib) [ 1540.717296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1540.746203] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1547.038719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1550.430724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.140@tcp [ 1550.794574] LNet: 94177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.801728] LNetError: 94177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.823480] LNet: Removed LNI 192.168.204.40@tcp [ 1551.195073] Key type .llcrypt unregistered [ 1551.199234] Key type ._llcrypt unregistered [ 1559.419343] Key type ._llcrypt registered [ 1559.423653] Key type .llcrypt registered [ 1560.409618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1561.694067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1563.013307] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:37:07 (1775043427) [ 1563.267824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.281792] alg: No test for adler32 (adler32-zlib) [ 1564.144566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.213270] LNet: 95264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.677386] Key type .llcrypt unregistered [ 1564.680310] Key type ._llcrypt unregistered [ 1568.761451] Key type ._llcrypt registered [ 1568.764266] Key type .llcrypt registered [ 1568.880642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1568.894235] alg: No test for adler32 (adler32-zlib) [ 1569.754590] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1569.758343] LNet: 95520:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1569.769526] LNet: 95520:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1569.789682] LNetError: 95532:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.42@tcp is being used as a gateway but routing feature is not turned on [ 1569.802161] LNetError: 95532:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.42@tcp has gone from up to down [ 1570.156638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.175125] LNet: 95578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.181942] LNetError: 95578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.197307] LNet: Removed LNI 192.168.204.40@tcp [ 1570.209773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.266282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1570.286391] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1570.289489] LNet: 95585:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1570.298107] LNet: 95585:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1570.306447] LNetError: 95585:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.204.42@tcp is a gateway. Can not delete it [ 1570.319218] LNetError: 95585:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.204.42@tcp is a router. Can not be deleted [ 1570.332831] LNet: 95585:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1570.333881] LNetError: 95582:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 1570.358627] LNetError: 95582:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1570.364709] LNetError: 95582:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.41@tcp has gone from up to down [ 1570.371314] LNetError: 95582:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1574.994652] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:37:19 (1775043439) [ 1576.303676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg440-server [ 1577.357817] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:37:21 (1775043441) [ 1578.488511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg440-server [ 1579.744683] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:37:24 (1775043444) [ 1580.679859] Lustre: Lustre: Build Version: 2.17.51_74_g79190eb [ 1582.432786] Key type lgssc registered [ 1583.298288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1596.619822] Key type lgssc unregistered [ 1596.887178] LNet: 98002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.896349] LNetError: 98002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.914239] LNet: Removed LNI 192.168.204.40@tcp [ 1597.307225] Key type .llcrypt unregistered [ 1597.310653] Key type ._llcrypt unregistered [ 1608.418526] Key type ._llcrypt registered [ 1608.422611] Key type .llcrypt registered [ 1609.400144] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:37:53 (1775043473) [ 1609.723280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1609.735554] alg: No test for adler32 (adler32-zlib) [ 1610.603481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.667814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1610.696934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1610.747798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1610.796723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1615.342677] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:37:59 (1775043479) [ 1615.582979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.604882] LNet: 99407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.625453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.692497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.720078] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1616.409866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.423803] LNet: 99551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.427863] LNetError: 99551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.440717] LNet: Removed LNI 192.168.204.40@tcp [ 1616.466173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.522418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1616.542715] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1617.001554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.033559] LNetError: 99579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.054712] LNet: Removed LNI 192.168.204.40@tcp [ 1617.073750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.136206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1617.180148] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1622.163521] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:38:06 (1775043486) [ 1622.350065] LNet: 100170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.357611] LNet: 100170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1622.362971] LNetError: 100170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.377175] LNet: Removed LNI 192.168.204.40@tcp [ 1622.965561] Key type .llcrypt unregistered [ 1622.968691] Key type ._llcrypt unregistered [ 1627.401982] Key type ._llcrypt registered [ 1627.403676] Key type .llcrypt registered [ 1627.555831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1627.571047] alg: No test for adler32 (adler32-zlib) [ 1628.449836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.510521] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1628.517894] LNet: *** cfs_fail_loc=e003, val=100*** [ 1629.093478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1629.114589] LNet: *** cfs_fail_loc=e003, val=100*** [ 1629.118804] LNet: Skipped 2 previous similar messages [ 1629.126655] Ping reply received with mapping: [ 1629.129156] found metadata reply NI[192.168.204.40@tcp] [ 1629.143949] 28 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1629.147200] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1629.153184] 5a 5a 5a 5a 5a 5a 5a 5a [ 1629.277922] LNet: 100567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.283313] LNetError: 100567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.298567] LNet: Removed LNI 192.168.204.40@tcp [ 1629.814638] Key type .llcrypt unregistered [ 1629.822526] Key type ._llcrypt unregistered [ 1638.178876] Key type ._llcrypt registered [ 1638.180928] Key type .llcrypt registered [ 1639.134350] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:38:23 (1775043503) [ 1639.724365] Key type .llcrypt unregistered [ 1639.730492] Key type ._llcrypt unregistered [ 1644.632328] Key type ._llcrypt registered [ 1644.634229] Key type .llcrypt registered [ 1644.787080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1644.798947] alg: No test for adler32 (adler32-zlib) [ 1645.856988] LNetError: 101612:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1646.109065] Key type .llcrypt unregistered [ 1646.114668] Key type ._llcrypt unregistered [ 1650.251814] Key type ._llcrypt registered [ 1650.257646] Key type .llcrypt registered [ 1651.313936] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:38:35 (1775043515) [ 1651.787667] Key type .llcrypt unregistered [ 1651.792340] Key type ._llcrypt unregistered [ 1656.288073] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1656.291881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1658.686652] Key type ._llcrypt registered [ 1658.689513] Key type .llcrypt registered [ 1658.853436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1658.863529] alg: No test for adler32 (adler32-zlib) [ 1659.756788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.830184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1666.364753] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:38:50 (1775043530) [ 1666.508020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.524257] LNet: 103391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.534203] LNetError: 103391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1666.553623] LNet: Removed LNI 10.1.2.3@tcp [ 1666.573606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.057881] Key type .llcrypt unregistered [ 1667.059605] Key type ._llcrypt unregistered [ 1670.312259] Key type ._llcrypt registered [ 1670.314780] Key type .llcrypt registered [ 1670.413615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1670.423927] alg: No test for adler32 (adler32-zlib) [ 1671.291319] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1671.329805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1671.437334] LNet: 103693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.443196] LNetError: 103693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.461388] LNet: Removed LNI 192.168.204.40@tcp [ 1671.871781] Key type .llcrypt unregistered [ 1671.873392] Key type ._llcrypt unregistered [ 1678.553265] Key type ._llcrypt registered [ 1678.557660] Key type .llcrypt registered [ 1679.455114] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:39:03 (1775043543) [ 1679.813317] Key type .llcrypt unregistered [ 1679.816220] Key type ._llcrypt unregistered [ 1683.445323] Key type ._llcrypt registered [ 1683.448409] Key type .llcrypt registered [ 1683.605654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1683.616825] alg: No test for adler32 (adler32-zlib) [ 1688.805997] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:39:13 (1775043553) [ 1688.948525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.972147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.023704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.042240] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1689.583284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.610326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1689.646150] LNet: Removed LNI 192.168.204.40@tcp [ 1689.826259] LNet: 105434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.110638] Key type .llcrypt unregistered [ 1690.113184] Key type ._llcrypt unregistered [ 1697.303424] Key type ._llcrypt registered [ 1697.306384] Key type .llcrypt registered [ 1698.161127] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:39:22 (1775043562) [ 1698.371746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.380673] alg: No test for adler32 (adler32-zlib) [ 1699.253137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.303156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.321272] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1699.931996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1699.962557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.40@tcp [ 1699.979839] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1699.985361] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1699.988388] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1699, 0, 0 [ 1699.993048] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1699.995657] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1699.998026] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1700.000227] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1699, 0, 0 [ 1700.011107] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1700.013488] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1700.016759] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1700.018649] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1699, 0, 1749 [ 1700.022227] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1700.024234] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1700.026402] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1700.029367] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1699, 0, 1749 [ 1700.032352] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1704.805254] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:39:28 (1775043568) [ 1705.087601] LNet: 106973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.093032] LNetError: 106973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.110961] LNet: Removed LNI 192.168.204.40@tcp [ 1705.560580] Key type .llcrypt unregistered [ 1705.561678] Key type ._llcrypt unregistered [ 1709.931328] Key type ._llcrypt registered [ 1709.933688] Key type .llcrypt registered [ 1710.107440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1710.123728] alg: No test for adler32 (adler32-zlib) [ 1710.997810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1711.023504] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1716.180979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1722.086787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.128972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1722.149519] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1722.951944] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1723.006642] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1723.012208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1723.017861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1725.323346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1725.356270] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1732.333579] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:39:56 (1775043596) [ 1732.409590] LNet: 108474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.414413] LNetError: 108474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.420306] LNet: Removed LNI 192.168.204.40@tcp [ 1733.472185] LNet: Removed LNI 192.168.204.40@tcp1 [ 1733.478972] LNetError: 108474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1733.482630] LNetError: 108474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1733.494125] LNet: Removed LNI 10.1.2.3@tcp99 [ 1733.899600] Key type .llcrypt unregistered [ 1733.902284] Key type ._llcrypt unregistered [ 1737.742798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1737.748941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1740.181389] Key type ._llcrypt registered [ 1740.184032] Key type .llcrypt registered [ 1740.347567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1740.357747] alg: No test for adler32 (adler32-zlib) [ 1741.232213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.268770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.291832] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1741.831957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1741.850591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1743.525162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1743.565418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1743.605515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1743.623792] LNetError: 109222:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1743.650910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1743.700954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1744.051649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1744.106169] LNet: 109242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.113214] LNetError: 109242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.123122] LNet: Removed LNI 192.168.204.40@tcp [ 1744.130582] LNet: Removed LNI 10.1.2.3@tcp [ 1744.663990] Key type .llcrypt unregistered [ 1744.669657] Key type ._llcrypt unregistered [ 1752.892440] Key type ._llcrypt registered [ 1752.894304] Key type .llcrypt registered [ 1753.791440] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:40:17 (1775043617) [ 1754.062455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.080092] alg: No test for adler32 (adler32-zlib) [ 1754.933377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.988448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.004963] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1755.551356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1755.623085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg440-client@tcp [ 1760.168366] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:40:24 (1775043624) [ 1760.270890] LNet: 110772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.282871] LNetError: 110772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.305844] LNet: Removed LNI 192.168.204.40@tcp [ 1760.811572] Key type .llcrypt unregistered [ 1760.814053] Key type ._llcrypt unregistered [ 1765.122785] Key type ._llcrypt registered [ 1765.125289] Key type .llcrypt registered [ 1765.250582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.265524] alg: No test for adler32 (adler32-zlib) [ 1766.129262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1766.144339] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1766.172221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1770.661141] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:40:34 (1775043634) [ 1771.959951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1773.217414] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:40:37 (1775043637) [ 1774.349523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1775.560393] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:40:39 (1775043639) [ 1775.627756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.648113] LNet: 112187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.654583] LNetError: 112187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.668754] LNet: Removed LNI 192.168.204.40@tcp [ 1775.686583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.736714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.751472] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1776.236756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1776.273946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1780.800346] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:40:45 (1775043645) [ 1780.868924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.891454] LNet: 112895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.897045] LNetError: 112895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.906073] LNet: Removed LNI 192.168.204.40@tcp [ 1780.921439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.972587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1780.999571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1781.436578] Key type .llcrypt unregistered [ 1781.438753] Key type ._llcrypt unregistered [ 1788.282275] Key type ._llcrypt registered [ 1788.283949] Key type .llcrypt registered [ 1789.016517] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:40:53 (1775043653) [ 1789.263744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.275152] alg: No test for adler32 (adler32-zlib) [ 1790.122134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.151355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.170980] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1790.580678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1790.628262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1794.507574] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:40:58 (1775043658) [ 1794.627738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.650567] LNet: 114456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.655642] LNetError: 114456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.669899] LNet: Removed LNI 192.168.204.40@tcp [ 1794.686597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.724169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1794.748883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1799.375918] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:41:03 (1775043663) [ 1799.478360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.500675] LNet: 115035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.534393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.592934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1799.945974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1799.986708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1800.048622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1800.363346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1800.388719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1800.440315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.466972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.514766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1800.789208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1800.809507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1800.853480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1801.151531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1801.193962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1805.120738] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:41:09 (1775043669) [ 1805.228182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.254764] LNet: 115670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.261023] LNet: 115670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1805.276450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.315852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.341678] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1805.841826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1805.863645] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1806.428358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1806.475952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1806.631245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.649562] LNetError: 115950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.656899] LNet: Removed LNI 192.168.204.40@tcp [ 1806.675448] LNet: Removed LNI 192.168.204.40@tcp2 [ 1806.706794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.757973] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1806.761629] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1806.765843] LNet: 115956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.018732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1807.174390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.224175] LNet: Removed LNI 192.168.204.40@tcp [ 1808.227009] LNetError: 115993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.232547] LNetError: 115993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1808.250229] LNet: Removed LNI 192.168.204.40@tcp2 [ 1808.272552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.309336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.327984] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1808.867762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1808.889603] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1809.481707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1809.508916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1809.543786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1809.717830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.739716] LNet: 116277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.748885] LNet: 116277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1810.784106] LNet: Removed LNI 192.168.204.40@tcp [ 1810.788752] LNetError: 116277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.796169] LNetError: 116277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1810.826658] LNet: Removed LNI 192.168.204.40@tcp2 [ 1810.849786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.905251] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1810.908521] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1810.911772] LNet: 116283:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1811.262800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1811.397408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.415209] LNet: Removed LNI 192.168.204.40@tcp [ 1812.457636] LNet: Removed LNI 192.168.204.40@tcp2 [ 1812.476942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.527652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.546767] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1812.989607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1813.004661] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1813.592726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1813.628424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1813.750445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.763585] LNet: 116600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.765960] LNet: 116600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1814.816176] LNet: Removed LNI 192.168.204.40@tcp [ 1814.818552] LNetError: 116600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.825432] LNetError: 116600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1814.833658] LNet: Removed LNI 192.168.204.40@tcp2 [ 1814.847394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.892229] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1814.897719] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1814.901509] LNet: 116606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1815.249792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1815.404209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.480590] LNet: Removed LNI 192.168.204.40@tcp [ 1816.490500] LNet: Removed LNI 192.168.204.40@tcp2 [ 1816.508381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.569945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.593443] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1817.036970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1817.051462] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1817.648638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1817.689301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1817.736849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1817.884703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.912615] LNet: Removed LNI 192.168.204.40@tcp [ 1819.948498] LNet: Removed LNI 192.168.204.40@tcp2 [ 1819.966212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.007055] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1820.009738] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1820.012268] LNet: 116934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1820.222441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1820.308249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.320311] LNet: Removed LNI 192.168.204.40@tcp [ 1821.352056] LNet: Removed LNI 192.168.204.40@tcp2 [ 1821.368907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.411505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.435550] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1821.854374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1821.872555] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1822.434450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1822.462081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1822.504464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1822.752714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.771138] LNet: 117267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.777422] LNet: 117267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1823.840200] LNet: Removed LNI 192.168.204.40@tcp [ 1823.843935] LNetError: 117267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.856302] LNetError: 117267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1823.868372] LNet: Removed LNI 192.168.204.40@tcp2 [ 1823.878875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.925581] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1823.934920] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1824.203363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1824.414511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.440349] LNet: Removed LNI 192.168.204.40@tcp [ 1825.454321] LNet: Removed LNI 192.168.204.40@tcp2 [ 1825.473900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.521712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.536837] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1826.012855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1826.028548] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1826.594917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1826.634278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1826.662403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1826.704900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1826.966232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.000158] LNet: Removed LNI 192.168.204.40@tcp [ 1829.033935] LNet: Removed LNI 192.168.204.40@tcp2 [ 1829.055801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.096902] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1829.103024] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1829.107857] LNet: 117633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.120581] LNet: 117633:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1829.428883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1833.518463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1834.633668] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:41:38 (1775043698) [ 1834.771652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.800961] LNet: Removed LNI 192.168.204.40@tcp [ 1834.812907] LNet: Removed LNI 192.168.204.40@tcp2 [ 1834.831332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.050635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.076337] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1835.582953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1835.601787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1836.254622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.274048] LNet: Removed LNI 192.168.204.40@tcp [ 1836.291114] LNet: Removed LNI 10.1.2.3@tcp [ 1836.314436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.360338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1836.375762] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1836.379634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1836.607724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1836.628486] LNet: Removed LNI 192.168.204.40@tcp [ 1836.643688] LNet: Removed LNI 10.1.2.3@tcp [ 1840.712986] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:41:45 (1775043705) [ 1840.769680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.793546] LNet: 119260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.800303] LNet: 119260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1840.818993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.903242] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1841.042192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1841.046213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1841.050858] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1843.282800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1843.301792] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1843.769233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1843.791521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1845.174783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1845.199634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1845.236633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.40@tcp -m PUT -e local_timeout -r 1 [ 1845.273583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1845.318176] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1845.321794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1845.460179] LNetError: 119324:0:(lib-move.c:4969:lnet_parse()) 192.168.204.40@tcp, src 192.168.204.40@tcp: Dropping PUT (error -108 looking up sender) [ 1846.370470] LNetError: 119783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.376024] LNetError: 119783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1846.384723] LNet: Removed LNI 192.168.204.40@tcp [ 1846.399056] LNet: Removed LNI 10.1.2.3@tcp [ 1846.642475] Key type .llcrypt unregistered [ 1846.644889] Key type ._llcrypt unregistered [ 1852.871060] Key type ._llcrypt registered [ 1852.874323] Key type .llcrypt registered [ 1853.646530] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:41:57 (1775043717) [ 1853.903931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1853.916669] alg: No test for adler32 (adler32-zlib) [ 1854.757948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.981723] LNet: 120635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.364495] Key type .llcrypt unregistered [ 1855.366638] Key type ._llcrypt unregistered [ 1858.962434] Key type ._llcrypt registered [ 1858.964475] Key type .llcrypt registered [ 1859.091706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1859.107325] alg: No test for adler32 (adler32-zlib) [ 1859.952574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1859.976034] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1865.107965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1865.231736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1865.618460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 1865.773796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1865.800537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1865.825297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1865.845926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1865.872421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1865.899437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1865.923547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.948250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1865.964805] LNet: 121083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1865.979807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1866.002547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1866.025332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1866.044350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1866.077919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1866.106774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.504670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1866.663929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1866.687121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1866.713971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1866.738361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1866.757596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1866.784482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1866.825776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1866.853694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1866.865243] LNet: 121201:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1866.877243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1866.896351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1866.919461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1866.940908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1866.970571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1867.001896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1867.553638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 1867.732816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1867.757142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1867.781512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1867.810882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1867.836246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1867.863952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1867.885136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1867.910667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1867.932436] LNet: 121318:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.959472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1867.986303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1868.023289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1868.047934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1868.070553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1868.091512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1868.509260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1868.679285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1868.708986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1868.732662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1868.754579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1868.776363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1868.798895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1868.825635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1868.850565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1868.873276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1868.904115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1868.930446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1868.956409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1868.990586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1869.026645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1869.224861] LNet: 121481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.228825] LNetError: 121481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.244202] LNet: Removed LNI 192.168.204.40@tcp [ 1869.479683] Key type .llcrypt unregistered [ 1869.482402] Key type ._llcrypt unregistered [ 1879.934073] Key type ._llcrypt registered [ 1879.935900] Key type .llcrypt registered [ 1880.788343] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:42:25 (1775043745) [ 1881.064355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.077654] alg: No test for adler32 (adler32-zlib) [ 1881.925946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.158207] LNet: 122367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.531554] Key type .llcrypt unregistered [ 1882.534661] Key type ._llcrypt unregistered [ 1886.204171] Key type ._llcrypt registered [ 1886.205858] Key type .llcrypt registered [ 1886.375769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1886.386748] alg: No test for adler32 (adler32-zlib) [ 1887.247567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1887.268558] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1891.974835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1896.896177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.931495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1896.945226] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1897.480727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1897.858278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 1897.997367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1898.022938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1898.045090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1898.064027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1898.091936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1898.115188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1898.136603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1898.163665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1898.174542] LNet: 123017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1898.189756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1898.208587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1898.230231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1898.254493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1898.273864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1898.296825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1898.645410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1898.787276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1898.807825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1898.831063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1898.851966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1898.876617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1898.900765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1898.931572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1898.958875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1898.971368] LNet: 123134:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1898.987634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1899.007688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1899.030171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1899.053903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1899.084028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1899.112915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1899.495878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 1899.552479] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1899.648366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1899.670763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1899.698776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1899.717201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1899.733700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1899.755486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1899.773818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1899.791739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1899.814646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1899.836580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1899.859387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1899.889958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1899.908353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1899.928223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1900.284164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1900.417062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1900.436103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1900.458953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1900.478944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1900.500711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1900.531332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1900.555542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1900.580192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1900.593654] LNet: 123369:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1900.599073] LNet: 123369:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1900.611877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1900.634427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1900.656800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1900.679379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1900.703855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1900.723763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1903.737933] LNet: 123449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.746722] LNetError: 123449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.752215] LNet: Removed LNI 192.168.204.40@tcp [ 1903.768280] LNet: Removed LNI 192.168.204.40@tcp1 [ 1903.994594] Key type .llcrypt unregistered [ 1903.999257] Key type ._llcrypt unregistered [ 1911.815488] Key type ._llcrypt registered [ 1911.821332] Key type .llcrypt registered [ 1912.612767] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:42:56 (1775043776) [ 1916.685404] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:43:01 (1775043781) [ 1920.851618] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:43:05 (1775043785) [ 1925.517352] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:43:09 (1775043789) [ 1929.893603] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:43:14 (1775043794) [ 1934.140435] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:43:18 (1775043798) [ 1938.123785] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:43:22 (1775043802) [ 1941.726364] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:43:26 (1775043806) [ 1945.545899] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:43:30 (1775043810) [ 1948.837223] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:43:33 (1775043813) [ 1952.055884] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:43:36 (1775043816) [ 1955.262966] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:43:39 (1775043819) [ 1958.950877] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:43:43 (1775043823) [ 1962.835848] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:43:47 (1775043827) [ 1966.437032] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:43:50 (1775043830) [ 1969.695746] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:43:54 (1775043834) [ 1972.967205] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:43:57 (1775043837) [ 1976.431495] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:44:00 (1775043840) [ 1976.624574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.635945] alg: No test for adler32 (adler32-zlib) [ 1977.477916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.539607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.554148] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1978.152069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.168324] LNet: 134148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.175147] LNetError: 134148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.185981] LNet: Removed LNI 192.168.204.40@tcp [ 1978.203248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.239634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1978.252609] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1981.604882] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:44:06 (1775043846) [ 1985.247446] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:44:09 (1775043849) [ 1989.006033] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:44:13 (1775043853) [ 1992.753443] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:44:17 (1775043857) [ 1996.129853] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:44:20 (1775043860) [ 1999.652417] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:44:24 (1775043864) [ 2002.947922] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:44:27 (1775043867) [ 2006.121370] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:44:30 (1775043870) [ 2009.286405] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:44:33 (1775043873) [ 2012.703371] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:44:37 (1775043877) [ 2016.141321] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:44:40 (1775043880) [ 2019.370390] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:44:43 (1775043883) [ 2022.477460] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1791 sec ========== 07:44:46 (1775043886) [ 2022.589492] LNet: 141604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.593349] LNetError: 141604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.608855] LNet: Removed LNI 192.168.204.40@tcp [ 2022.921359] Key type .llcrypt unregistered [ 2022.923070] Key type ._llcrypt unregistered [ 2028.129643] Key type ._llcrypt registered [ 2028.131263] Key type .llcrypt registered [ 2028.340891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2028.347791] alg: No test for adler32 (adler32-zlib) [ 2029.263518] Lustre: Lustre: Build Version: 2.17.51_74_g79190eb [ 2029.381414] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2030.992202] Key type lgssc registered [ 2031.501122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.037296] Lustre: Mounted lustre-client [ 2066.229816] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2075.015203] LustreError: 144047:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff94591a0ae000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2075.042203] Lustre: Unmounted lustre-client [ 2085.647241] Key type lgssc unregistered [ 2085.803881] LNet: 144526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.808969] LNetError: 144526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.828058] LNet: Removed LNI 192.168.204.40@tcp [ 2086.155171] Key type .llcrypt unregistered [ 2086.157093] Key type ._llcrypt unregistered