[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438774880 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006786] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010071] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.013005] *** VALIDATE selinux *** [ 0.021301] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026094] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] *** VALIDATE tmpfs *** [ 0.031006] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041359] debug: unmapping init [mem 0xffffffff8d859000-0xffffffff8d860fff] [ 0.043881] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046017] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.053255] rcu: Hierarchical SRCU implementation. [ 0.055309] smp: Bringing up secondary CPUs ... [ 0.056592] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060693] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.260618] node 0 deferred pages initialised in 194ms [ 0.265106] devtmpfs: initialized [ 0.266305] x86/mm: Memory block size: 128MB [ 0.269844] gcov: version magic: 0x41383552 [ 0.273283] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.274087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275264] pinctrl core: initialized pinctrl subsystem [ 0.276186] [ 0.276679] ************************************************************* [ 0.277017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278015] ** ** [ 0.279017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.280013] ** ** [ 0.281013] ** This means that this kernel is built to expose internal ** [ 0.282015] ** IOMMU data structures, which may compromise security on ** [ 0.283015] ** your system. ** [ 0.284016] ** ** [ 0.285017] ** If you see this message and you are not debugging the ** [ 0.286015] ** kernel, report this immediately to your vendor! ** [ 0.287011] ** ** [ 0.288014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289016] ************************************************************* [ 0.290624] NET: Registered protocol family 16 [ 0.292518] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.299091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303029] cpuidle: using governor menu [ 0.305116] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.308572] PCI: Using configuration type 1 for base access [ 0.310133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326069] cryptd: max_cpu_qlen set to 1000 [ 0.330033] ACPI: Added _OSI(Module Device) [ 0.331024] ACPI: Added _OSI(Processor Device) [ 0.333025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.343076] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.348806] ACPI: Interpreter enabled [ 0.350140] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352021] ACPI: Using IOAPIC for interrupt routing [ 0.354141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.357467] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370402] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.372067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.375030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384000] acpiphp: Slot [2] registered [ 0.385047] acpiphp: Slot [5] registered [ 0.386140] acpiphp: Slot [6] registered [ 0.388138] acpiphp: Slot [3] registered [ 0.389154] acpiphp: Slot [4] registered [ 0.390154] acpiphp: Slot [7] registered [ 0.391132] acpiphp: Slot [8] registered [ 0.393132] acpiphp: Slot [9] registered [ 0.394149] acpiphp: Slot [10] registered [ 0.396144] acpiphp: Slot [11] registered [ 0.398183] acpiphp: Slot [12] registered [ 0.399151] acpiphp: Slot [13] registered [ 0.401146] acpiphp: Slot [14] registered [ 0.402202] acpiphp: Slot [15] registered [ 0.404186] acpiphp: Slot [16] registered [ 0.405135] acpiphp: Slot [17] registered [ 0.407193] acpiphp: Slot [18] registered [ 0.408173] acpiphp: Slot [19] registered [ 0.410204] acpiphp: Slot [20] registered [ 0.411000] acpiphp: Slot [21] registered [ 0.411000] acpiphp: Slot [22] registered [ 0.413115] acpiphp: Slot [23] registered [ 0.414117] acpiphp: Slot [24] registered [ 0.416099] acpiphp: Slot [25] registered [ 0.417091] acpiphp: Slot [26] registered [ 0.419102] acpiphp: Slot [27] registered [ 0.421176] acpiphp: Slot [28] registered [ 0.422119] acpiphp: Slot [29] registered [ 0.424096] acpiphp: Slot [30] registered [ 0.425129] acpiphp: Slot [31] registered [ 0.427076] PCI host bridge to bus 0000:00 [ 0.428021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.430021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.433021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.435020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.437021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.439024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.441189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.448338] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.455015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.459069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.465018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.470913] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.473762] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.479781] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.495726] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.501014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.507808] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.512944] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.519014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.533955] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543505] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.552018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.559014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.575018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.587583] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.590495] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.594510] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.598690] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.600326] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.607131] iommu: Default domain type: Passthrough [ 0.609469] SCSI subsystem initialized [ 0.611127] ACPI: bus type USB registered [ 0.613154] usbcore: registered new interface driver usbfs [ 0.615110] usbcore: registered new interface driver hub [ 0.618087] usbcore: registered new device driver usb [ 0.620197] pps_core: LinuxPPS API ver. 1 registered [ 0.622011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.625050] PTP clock support registered [ 0.627095] EDAC MC: Ver: 3.0.0 [ 0.628297] PCI: Using ACPI for IRQ routing [ 0.630813] NetLabel: Initializing [ 0.632012] NetLabel: domain hash size = 128 [ 0.634013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.636086] NetLabel: unlabeled traffic allowed by default [ 0.638083] vgaarb: loaded [ 0.639234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.641014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.649396] clocksource: Switched to clocksource kvm-clock [ 0.754392] VFS: Disk quotas dquot_6.6.0 [ 0.756312] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759080] *** VALIDATE ramfs *** [ 0.760754] *** VALIDATE hugetlbfs *** [ 0.762551] pnp: PnP ACPI init [ 0.765702] pnp: PnP ACPI: found 6 devices [ 0.786237] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.789691] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.792157] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.794981] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.798151] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.801155] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.804660] NET: Registered protocol family 2 [ 0.807802] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.813638] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.818074] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.824334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.828417] TCP: Hash tables configured (established 65536 bind 65536) [ 0.831915] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.835807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839191] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.842939] NET: Registered protocol family 1 [ 0.846439] RPC: Registered named UNIX socket transport module. [ 0.849186] RPC: Registered udp transport module. [ 0.851179] RPC: Registered tcp transport module. [ 0.853273] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.874065] NET: Registered protocol family 44 [ 0.878604] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.880603] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.882685] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.884926] PCI: CLS 0 bytes, default 64 [ 0.889221] Unpacking initramfs... [ 2.289607] debug: unmapping init [mem 0xffff93de3cc64000-0xffff93de3ffcffff] [ 2.293661] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.296404] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.299328] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.782077] Initialise system trusted keyrings [ 2.784230] Key type blacklist registered [ 2.786402] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.795866] zbud: loaded [ 2.799560] *** VALIDATE nfs *** [ 2.800961] *** VALIDATE nfs4 *** [ 2.802611] pstore: using deflate compression [ 2.805824] Platform Keyring initialized [ 2.906651] NET: Registered protocol family 38 [ 2.910375] Key type asymmetric registered [ 2.912218] Asymmetric key parser 'x509' registered [ 2.917847] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.921421] io scheduler mq-deadline registered [ 2.923236] io scheduler kyber registered [ 2.925135] io scheduler bfq registered [ 2.927477] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.930619] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.933547] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.935723] ACPI: Power Button [PWRF] [ 2.939862] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.946736] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.960453] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.989915] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.019917] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.026506] Non-volatile memory driver v1.3 [ 3.028165] Linux agpgart interface v0.103 [ 3.056893] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.060246] vda: detected capacity change from 0 to 74698752 [ 3.083500] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.087408] vdb: detected capacity change from 0 to 1073741824 [ 3.104153] libphy: Fixed MDIO Bus: probed [ 3.109336] usbcore: registered new interface driver usbserial_generic [ 3.112033] usbserial: USB Serial support registered for generic [ 3.114924] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.119378] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.121434] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.124108] mousedev: PS/2 mouse device common for all mice [ 3.127214] rtc_cmos 00:05: RTC can wake from S4 [ 3.129381] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.131399] rtc_cmos 00:05: registered as rtc0 [ 3.135055] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.135861] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.138901] intel_pstate: CPU model not supported [ 3.142837] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.148675] hid: raw HID events driver (C) Jiri Kosina [ 3.151097] usbcore: registered new interface driver usbhid [ 3.153424] usbhid: USB HID core driver [ 3.155333] drop_monitor: Initializing network drop monitor service [ 3.158565] Initializing XFRM netlink socket [ 3.160567] NET: Registered protocol family 10 [ 3.163453] Segment Routing with IPv6 [ 3.165012] NET: Registered protocol family 17 [ 3.167101] mpls_gso: MPLS GSO support [ 3.173641] RAS: Correctable Errors collector initialized. [ 3.176051] AVX version of gcm_enc/dec engaged. [ 3.177683] AES CTR mode by8 optimization enabled [ 3.247298] sched_clock: Marking stable (3247029733, 0)->(4220446598, -973416865) [ 3.250678] registered taskstats version 1 [ 3.252220] Loading compiled-in X.509 certificates [ 3.253889] zswap: loaded using pool lzo/zbud [ 3.279984] Key type big_key registered [ 3.291195] Key type encrypted registered [ 3.292453] ima: No TPM chip found, activating TPM-bypass! [ 3.294818] ima: Allocated hash algorithm: sha1 [ 3.296768] ima: No architecture policies found [ 3.298691] evm: Initialising EVM extended attributes: [ 3.300705] evm: security.selinux [ 3.301942] evm: security.ima [ 3.303120] evm: security.capability [ 3.304572] evm: HMAC attrs: 0x1 [ 3.306847] rtc_cmos 00:05: setting system clock to 2026-08-14 12:07:00 UTC (1786709220) [ 3.312954] debug: unmapping init [mem 0xffffffff8e803000-0xffffffff8e9fffff] [ 3.315954] debug: unmapping init [mem 0xffffffff8d582000-0xffffffff8d858fff] [ 3.328140] Write protecting the kernel read-only data: 28672k [ 3.332421] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 3.335603] debug: unmapping init [mem 0xffffffff8c514000-0xffffffff8c5fffff] [ 3.369718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.378410] systemd[1]: Detected virtualization kvm. [ 3.381899] systemd[1]: Detected architecture x86-64. [ 3.383704] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.406498] systemd[1]: No hostname configured. [ 3.408193] systemd[1]: Set hostname to . [ 3.410528] random: systemd: uninitialized urandom read (16 bytes read) [ 3.413250] systemd[1]: Initializing machine ID from random generator. [ 3.527595] random: systemd: uninitialized urandom read (16 bytes read) [ 3.530574] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.535756] random: systemd: uninitialized urandom read (16 bytes read) [ 3.538333] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.544158] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.070053] device-mapper: uevent: version 1.0.3 [ 4.072143] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.740166] virtio_net virtio0 ens2: renamed from eth0 [ 4.763585] scsi host0: ata_piix [ 4.765947] random: fast init done [ 4.769359] scsi host1: ata_piix [ 4.771695] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.774769] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.426944] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.808951] random: crng init done [ 9.810560] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.100560] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.168719] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.426366] SELinux: Disabled at runtime. [ 11.487869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.496476] systemd[1]: Detected virtualization kvm. [ 11.498395] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.963593] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.966279] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.972697] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.977139] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.981114] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.987973] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.991520] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.132187] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.383060] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.655784] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.683189] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.769078] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.791908] EDAC sbridge: Ver: 1.1.2 [ 13.748450] Key type dns_resolver registered [ 14.055206] NFS: Registering the id_resolver key type [ 14.057120] Key type id_resolver registered [ 14.058615] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg352-client login: [ 56.306208] hrtimer: interrupt took 3432119 ns [ 69.594891] libcfs: loading out-of-tree module taints kernel. [ 69.725278] Key type ._llcrypt registered [ 69.741408] Key type .llcrypt registered [ 70.246936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 70.266479] alg: No test for adler32 (adler32-zlib) [ 71.842955] Lustre: Lustre: Build Version: 2.17.57_1_g92903bd [ 72.546263] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 74.311337] Key type lgssc registered [ 76.435783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 213.173731] Lustre: Mounted lustre-client - version 2.17.57_1_g92903bd [ 218.635769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.162984] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing check_logdir /tmp/testlogs/ [ 239.075872] Lustre: lustre-OST0000-osc-ffff93de8824b800: disconnect after 24s idle [ 239.410689] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing yml_node [ 244.326700] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 246.719082] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 248.947395] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 250.764687] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 08:11:06 EDT 2026 [ 267.440443] Lustre: Unmounted lustre-client [ 305.237851] Key type lgssc unregistered [ 305.423995] LNet: 8406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.433397] LNetError: 8406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.447842] LNet: Removed LNI 192.168.203.52@tcp [ 306.040581] Key type .llcrypt unregistered [ 306.043711] Key type ._llcrypt unregistered [ 336.611466] Key type ._llcrypt registered [ 336.613745] Key type .llcrypt registered [ 336.762537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.776386] alg: No test for adler32 (adler32-zlib) [ 337.696301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 337.749559] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 337.773120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 341.867118] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 342.142767] LNet: 9296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.150958] LNetError: 9296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.169909] LNet: Removed LNI 192.168.203.52@tcp [ 342.884514] Key type .llcrypt unregistered [ 342.891185] Key type ._llcrypt unregistered [ 349.299770] Key type ._llcrypt registered [ 349.302200] Key type .llcrypt registered [ 350.704847] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:12:46 (1786709566) [ 351.271855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.292821] alg: No test for adler32 (adler32-zlib) [ 352.178618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.257589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.296814] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.319866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.388642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.435689] LNet: 9719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 359.370435] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:12:55 (1786709575) [ 359.668785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.701639] LNet: 10298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.727794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.837646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 366.599583] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:13:02 (1786709582) [ 366.976581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.024277] LNet: 10884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.055742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.119298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 373.400122] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:13:09 (1786709589) [ 373.479897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.508319] LNet: 11462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.530956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.688546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 383.028366] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:13:18 (1786709598) [ 383.170755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.207659] LNet: 12048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.253990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.364600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 384.193189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 392.705430] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:13:28 (1786709608) [ 392.834182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.883924] LNet: 12639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.937784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.107369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 393.807189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 394.515934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 401.871826] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:13:37 (1786709617) [ 402.318331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.373077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.587804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 410.190418] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:13:46 (1786709626) [ 410.354672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.395121] LNet: 13827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.405973] LNet: 13827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 410.438839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.507180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 410.546837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 411.063285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 411.116299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 411.579326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 411.615571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 412.153813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 412.184778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 412.557758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 412.602960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 413.165950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 413.212078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.798703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 413.836106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 414.386087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 414.441556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 414.991125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 415.025057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 415.502189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 415.542901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 416.069211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 416.114924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 416.533788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 416.586783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 416.996586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.038259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.921680] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:13:59 (1786709639) [ 424.373950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.424584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.483579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 424.532598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 431.682720] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:14:07 (1786709647) [ 432.102803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.151976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.270923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 432.349454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 439.426345] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:14:15 (1786709655) [ 439.969859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.040959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.183936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 440.246628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 446.564865] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:14:22 (1786709662) [ 446.652981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.691380] LNet: 16262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.700706] LNet: 16262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 446.730737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.840465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 446.882681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.284225] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:14:29 (1786709669) [ 453.473960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.513558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.564616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 453.596664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 460.019523] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:14:35 (1786709675) [ 460.154993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.255702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.362386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.406577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.523132] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:14:44 (1786709684) [ 468.982832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.052598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.201740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.791366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.675377] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:14:53 (1786709693) [ 477.775062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.842854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.198832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.458445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.859036] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:15:01 (1786709701) [ 486.059892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.150635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.327318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.615336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.598882] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:15:09 (1786709709) [ 494.013084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.069690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.641916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.937244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.842508] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:15:17 (1786709717) [ 502.197503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.265864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.349799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.440165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.562909] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:15:25 (1786709725) [ 509.781862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.841675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.933890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.026067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.670399] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:15:32 (1786709732) [ 516.994980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.047198] LNet: 21710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.055995] LNet: 21710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 517.083245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.166571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 524.368513] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:15:40 (1786709740) [ 524.649680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.689604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.781654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 532.254783] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:15:48 (1786709748) [ 532.482366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.518664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.610376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 533.078859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 540.441424] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:15:56 (1786709756) [ 540.595629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.646172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.726936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 540.791707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 548.500350] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:16:04 (1786709764) [ 548.636744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.756766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.902506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 548.992790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 556.403686] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:16:12 (1786709772) [ 556.632706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.681883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.815580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 556.897555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 563.976669] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:16:19 (1786709779) [ 564.252047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.312667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.414906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.454852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.700510] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:16:27 (1786709787) [ 571.815531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.845604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.911427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 571.977651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 572.001091] LNetError: 25827:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 572.051645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 572.082692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 578.485559] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:16:34 (1786709794) [ 578.836905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.897908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.037855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9iU20BgQp9 [ 579.144327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9iU20BgQp9 [ 579.197938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9iU20BgQp9 [ 586.124339] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:16:41 (1786709801) [ 586.352465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.378834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.424208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 586.458568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 586.497800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 593.463142] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:16:49 (1786709809) [ 593.767243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.802802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.860813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.885445] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 595.199719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 596.698448] Lustre: Lustre: Build Version: 2.17.57_1_g92903bd [ 598.696195] Key type lgssc registered [ 600.180958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 604.067305] LNetError: 27588:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.52@tcp99 has no available nets [ 606.988644] Key type lgssc unregistered [ 607.384390] LNetError: 28698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.411504] LNet: Removed LNI 192.168.203.52@tcp [ 608.115438] Key type .llcrypt unregistered [ 608.119173] Key type ._llcrypt unregistered [ 615.826339] Key type ._llcrypt registered [ 615.832346] Key type .llcrypt registered [ 617.685880] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:17:12 (1786709832) [ 618.224286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.243355] alg: No test for adler32 (adler32-zlib) [ 619.175164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.237401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.465399] LNet: Message forwarding will be enabled in 2s [ 625.567129] LNet: Message forwarding enabled [ 625.907869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.930316] LNet: Message forwarding will stop in 2s to 6s [ 630.660451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 631.723318] LNet: Message forwarding disabled [ 632.852792] LNet: Message forwarding will be enabled in 2s [ 635.807101] LNet: Message forwarding enabled [ 640.332614] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:17:36 (1786709856) [ 640.605891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.634821] LNet: 30105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.123920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.216701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 649.317893] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:17:44 (1786709864) [ 649.424290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.465884] LNet: 30682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.500657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.648340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 649.676364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 649.712067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 649.754205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 649.800393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 649.853360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 649.905786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 649.970848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 650.049345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 650.113901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 650.178802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 650.236400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 650.274903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 650.337352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 650.401787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 650.457483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 650.507378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 650.578617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 650.629491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 650.662094] LNetError: 30730:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 658.172720] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:17:53 (1786709873) [ 658.384917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.417921] LNet: 31301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.450356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.512825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.845952] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:18:00 (1786709880) [ 666.541941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg352-server [ 668.270252] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:18:04 (1786709884) [ 668.581840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.609434] LNet: 32233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.664216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.707188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.738220] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 669.552426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[53-57]@tcp [ 669.575638] LNet: 32366:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 669.623322] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 669.649610] LNetError: 32237:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.55@tcp has gone from up to down [ 670.648771] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 670.666132] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 671.670980] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 671.691491] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 673.700364] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.54@tcp is being used as a gateway but routing feature is not turned on [ 673.734956] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 676.202860] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:18:12 (1786709892) [ 677.802774] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 677.828535] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 678.185623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg352-server [ 680.289841] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:18:15 (1786709895) [ 680.409775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.427880] LNet: 33296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.433194] LNetError: 33296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.461388] LNet: Removed LNI 192.168.203.52@tcp [ 680.524821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.599115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.655365] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 681.406529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[53-58/2]@tcp [ 681.427487] LNet: 33419:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 681.440887] LNet: 33419:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 681.465306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[53-58/2]@tcp [ 681.484712] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.57@tcp has gone from up to down [ 681.526251] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 688.863427] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:18:24 (1786709904) [ 689.098522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.128716] LNet: 34012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.136477] LNetError: 34012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.157452] LNet: Removed LNI 192.168.203.52@tcp [ 689.205675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.342907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 689.420713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.499660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.574509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.644213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.690932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.771677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 689.860414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.898357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.984524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.058235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.099274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.151893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 690.281709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.312028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.384730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.451936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.488383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.554403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 690.649507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.706632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.782402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.853553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.898267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.930871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 690.990913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.020461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.072270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.183268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.237532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.319027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 691.401238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.446400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.523405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.473907] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:18:33 (1786709913) [ 699.496340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg352-server [ 700.999159] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:18:36 (1786709916) [ 703.199434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg352-server [ 705.041595] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:18:40 (1786709920) [ 705.461600] LNet: 35415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.473370] LNet: 35415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 706.365341] Key type .llcrypt unregistered [ 706.368697] Key type ._llcrypt unregistered [ 714.064983] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.314662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.323292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.329273] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.892811] Key type ._llcrypt registered [ 716.894853] Key type .llcrypt registered [ 717.059584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.071214] alg: No test for adler32 (adler32-zlib) [ 718.027415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.084828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.113554] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 718.865577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.890783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.728786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 720.750912] LNet: Removed LNI 10.1.2.3@tcp [ 720.899306] LNet: 36136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.907952] LNetError: 36136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.924704] LNet: Removed LNI 192.168.203.52@tcp [ 721.689732] Key type .llcrypt unregistered [ 721.694560] Key type ._llcrypt unregistered [ 734.854301] Key type ._llcrypt registered [ 734.860270] Key type .llcrypt registered [ 736.514067] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:19:11 (1786709951) [ 737.154467] Key type .llcrypt unregistered [ 737.166923] Key type ._llcrypt unregistered [ 744.403839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 744.628991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 744.643952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 744.646495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 747.303355] Key type ._llcrypt registered [ 747.305895] Key type .llcrypt registered [ 747.554063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.594708] alg: No test for adler32 (adler32-zlib) [ 748.637350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.744711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.804777] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 749.604176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 749.632488] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.562610] LNet: 37640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.575794] LNetError: 37640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 752.596330] LNet: Removed LNI 10.1.2.3@tcp [ 752.622361] LNet: Removed LNI 192.168.203.52@tcp [ 753.093332] Key type .llcrypt unregistered [ 753.096467] Key type ._llcrypt unregistered [ 765.934996] Key type ._llcrypt registered [ 765.939067] Key type .llcrypt registered [ 767.340735] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:19:42 (1786709982) [ 768.070407] Key type .llcrypt unregistered [ 768.073954] Key type ._llcrypt unregistered [ 775.218740] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 775.341846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 775.350459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 775.359035] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.060794] Key type ._llcrypt registered [ 778.063344] Key type .llcrypt registered [ 778.232993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.258560] alg: No test for adler32 (adler32-zlib) [ 779.202490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.261916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 779.306398] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 781.287760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.327191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.723266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 783.749222] LNet: Removed LNI 10.1.2.3@tcp [ 783.766364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 783.794809] LNet: Removed LNI 10.1.2.31@tcp [ 783.814820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.843865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.696723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 784.721343] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.714862] LNet: 39571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.722737] LNetError: 39571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 785.730605] LNet: Removed LNI 10.1.2.31@tcp [ 785.741746] LNet: Removed LNI 10.1.2.3@tcp [ 786.319942] Key type .llcrypt unregistered [ 786.321486] Key type ._llcrypt unregistered [ 801.750794] Key type ._llcrypt registered [ 801.752292] Key type .llcrypt registered [ 803.447407] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:20:18 (1786710018) [ 803.791853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.803279] alg: No test for adler32 (adler32-zlib) [ 804.785487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.889819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.942682] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 805.908804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.948486] LNet: 40528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.959449] LNetError: 40528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.993260] LNet: Removed LNI 192.168.203.52@tcp [ 806.025117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.110256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.52@tcp [ 806.156404] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 807.214206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.52@tcp [ 807.255160] LNet: Removed LNI 192.168.203.52@tcp [ 807.357907] LNet: 40663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.455844] Key type .llcrypt unregistered [ 808.461885] Key type ._llcrypt unregistered [ 822.901200] Key type ._llcrypt registered [ 822.906503] Key type .llcrypt registered [ 824.167382] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:20:39 (1786710039) [ 826.397092] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg352-server [ 828.281621] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:20:44 (1786710044) [ 828.870990] Key type .llcrypt unregistered [ 828.874438] Key type ._llcrypt unregistered [ 836.310882] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.557217] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.562245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.574553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.252603] Key type ._llcrypt registered [ 839.254867] Key type .llcrypt registered [ 839.462211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.479547] alg: No test for adler32 (adler32-zlib) [ 840.492121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.600239] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 841.322395] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 844.009644] LNet: 42509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.014288] LNetError: 42509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.020696] LNet: Removed LNI 192.168.203.52@tcp1 [ 844.029862] LNet: Removed LNI 10.1.2.3@tcp2 [ 844.706918] Key type .llcrypt unregistered [ 844.710331] Key type ._llcrypt unregistered [ 858.715832] Key type ._llcrypt registered [ 858.729159] Key type .llcrypt registered [ 860.233338] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:21:15 (1786710075) [ 860.922606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.943955] alg: No test for adler32 (adler32-zlib) [ 861.869842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.926972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.961710] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 862.662086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.683419] LNet: 43483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.691833] LNetError: 43483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.706993] LNet: Removed LNI 192.168.203.52@tcp [ 862.732390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.784979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 862.814578] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 870.275387] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:21:25 (1786710085) [ 870.533566] LNet: 44074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.545045] LNetError: 44074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.560802] LNet: Removed LNI 192.168.203.52@tcp [ 871.224491] Key type .llcrypt unregistered [ 871.236567] Key type ._llcrypt unregistered [ 877.515569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.729535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.738578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.745804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.254645] Key type ._llcrypt registered [ 880.256803] Key type .llcrypt registered [ 880.432414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.468186] alg: No test for adler32 (adler32-zlib) [ 881.374752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.444874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.497227] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 882.242398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 882.272147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.494949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.538192] LNet: 44830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.551784] LNetError: 44830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.566752] LNet: Removed LNI 10.1.2.3@tcp [ 884.583025] LNet: Removed LNI 192.168.203.52@tcp [ 884.650805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.720667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.750606] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 884.758857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.376341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.401531] LNet: 44857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.406968] LNetError: 44857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.412498] LNetError: 44857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.416748] LNet: Removed LNI 10.1.2.3@tcp [ 885.424980] LNet: Removed LNI 192.168.203.52@tcp [ 885.476570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.550141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.602897] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 885.615935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.138990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 886.155966] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 887.198948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.216611] LNet: 45019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.222792] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 887.232523] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.243436] LNet: Removed LNI 10.1.2.3@tcp [ 887.245884] LNet: Removed LNI 192.168.203.52@tcp [ 887.271181] LNet: Removed LNI 192.168.203.52@tcp2 [ 887.326615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.378730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 887.414980] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 887.421254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 887.426205] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 894.607517] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:21:50 (1786710110) [ 894.899804] LNet: 45617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.905211] LNetError: 45617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.912319] LNetError: 45617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 894.916889] LNet: Removed LNI 10.1.2.3@tcp [ 894.920585] LNet: Removed LNI 192.168.203.52@tcp [ 894.926142] LNet: Removed LNI 192.168.203.52@tcp2 [ 895.506562] Key type .llcrypt unregistered [ 895.513135] Key type ._llcrypt unregistered [ 902.158460] Key type ._llcrypt registered [ 902.165737] Key type .llcrypt registered [ 902.381893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.400733] alg: No test for adler32 (adler32-zlib) [ 903.325981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.390210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.413464] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 904.151347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.173128] LNet: 46033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.187735] LNetError: 46033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.210357] LNet: Removed LNI 192.168.203.52@tcp [ 904.243969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.326915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.367824] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 904.726508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.758225] LNet: 46063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.762893] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.773438] LNet: Removed LNI 192.168.203.52@tcp [ 904.812524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.899663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.945798] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 905.441806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.470965] LNet: Removed LNI 192.168.203.52@tcp [ 905.498229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.562755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.596103] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 906.097735] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.249355] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.258236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.272343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 908.607336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.633529] LNet: 46174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.639514] LNet: 46174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 908.645589] LNetError: 46174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.650289] LNetError: 46174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 908.668487] LNet: Removed LNI 192.168.203.52@tcp [ 908.689272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.743903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.771487] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 915.493620] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:22:11 (1786710131) [ 915.905133] LNet: 46772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.913242] LNetError: 46772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.933395] LNet: Removed LNI 192.168.203.52@tcp [ 916.627657] Key type .llcrypt unregistered [ 916.634472] Key type ._llcrypt unregistered [ 923.454932] Key type ._llcrypt registered [ 923.457809] Key type .llcrypt registered [ 923.631199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.644074] alg: No test for adler32 (adler32-zlib) [ 924.524265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.568214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.588299] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 925.434199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.454635] LNet: 47174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.459905] LNetError: 47174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.479193] LNet: Removed LNI 192.168.203.52@tcp [ 925.493062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.550934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 925.576799] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 931.754312] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:22:27 (1786710147) [ 931.944646] LNet: 47766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.954321] LNetError: 47766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.980077] LNet: Removed LNI 192.168.203.52@tcp [ 932.696824] Key type .llcrypt unregistered [ 932.699494] Key type ._llcrypt unregistered [ 939.248987] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.448430] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.461317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.465308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.021412] Key type ._llcrypt registered [ 942.024400] Key type .llcrypt registered [ 942.196206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.216627] alg: No test for adler32 (adler32-zlib) [ 943.132974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.192694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.218859] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 944.153890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.193784] LNet: 48220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.201376] LNetError: 48220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.229986] LNet: Removed LNI 192.168.203.52@tcp [ 944.268449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.363226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 944.401966] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 951.579946] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:22:47 (1786710167) [ 951.948718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.009676] LNet: 48819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.021466] LNetError: 48819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.033524] LNet: Removed LNI 192.168.203.52@tcp [ 952.081023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.227204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 958.893255] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:22:54 (1786710174) [ 959.045585] LNet: 49397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.868887] Key type .llcrypt unregistered [ 959.875985] Key type ._llcrypt unregistered [ 966.590616] Key type ._llcrypt registered [ 966.592987] Key type .llcrypt registered [ 966.809385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.829760] alg: No test for adler32 (adler32-zlib) [ 967.722746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.772521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.792943] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 968.583388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.616663] LNet: 49803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.626669] LNetError: 49803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.656734] LNet: Removed LNI 192.168.203.52@tcp [ 968.701991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.792400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 968.816776] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 976.507589] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:23:12 (1786710192) [ 976.626800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.663554] LNet: 50390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.672562] LNetError: 50390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.697523] LNet: Removed LNI 192.168.203.52@tcp [ 976.739177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.878183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 983.137401] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:23:18 (1786710198) [ 984.702400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 986.209661] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:23:22 (1786710202) [ 988.051679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 989.731627] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:23:25 (1786710205) [ 991.584397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 993.323780] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:23:29 (1786710209) [ 994.677161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 996.405429] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:23:32 (1786710212) [ 998.050914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 999.781613] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:23:35 (1786710215) [ 1000.046209] LNet: 52450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.735682] Key type .llcrypt unregistered [ 1000.738341] Key type ._llcrypt unregistered [ 1007.139313] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1007.346614] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1007.363188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.370697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.883893] Key type ._llcrypt registered [ 1009.887219] Key type .llcrypt registered [ 1010.027125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.036574] alg: No test for adler32 (adler32-zlib) [ 1010.948153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.033771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1011.060212] LNet: Added LNI 192.168.203.52@tcp [8/512/0/180] [ 1011.069396] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1011.072318] LNet: 52775:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1011.164331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.209423] LNet: 52794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.223853] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.243232] LNet: Removed LNI 10.1.2.3@tcp [ 1011.248520] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.273657] LNet: Removed LNI 192.168.203.52@tcp [ 1011.293464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.375336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1011.413933] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1011.500976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.526489] LNet: 52818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.532786] LNetError: 52818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.544862] LNet: Removed LNI 192.168.203.52@tcp [ 1011.560936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.609964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1011.638629] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1011.643120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1011.645860] LNet: 52826:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1011.729869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.759608] LNet: 52844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.769529] LNetError: 52844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.776900] LNet: Removed LNI 10.1.2.3@tcp [ 1011.778567] LNetError: 52844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.790842] LNet: Removed LNI 192.168.203.52@tcp [ 1011.813124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.895625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1011.918664] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1011.922518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1011.925238] LNet: 52852:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1012.098780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.124884] LNet: 52878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.132251] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.139584] LNet: Removed LNI 10.1.2.3@tcp [ 1012.141885] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.176303] LNet: Removed LNI 192.168.203.52@tcp [ 1012.205110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.280408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1012.349119] LNet: Added LNI 192.168.203.52@tcp [8/256/0/182] [ 1012.362929] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1012.497975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.513588] LNet: 52907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.521198] LNetError: 52907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.525115] LNet: Removed LNI 10.1.2.3@tcp [ 1012.527818] LNetError: 52907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.546515] LNet: Removed LNI 192.168.203.52@tcp [ 1012.562897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.615619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1012.633749] LNet: Added LNI 192.168.203.52@tcp [8/256/0/182] [ 1012.638167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1012.640203] LNet: 52915:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1012.731768] LNet: Removed LNI 10.1.2.3@tcp [ 1012.745715] LNet: Removed LNI 192.168.203.52@tcp [ 1013.394352] Key type .llcrypt unregistered [ 1013.398106] Key type ._llcrypt unregistered [ 1024.098182] Key type ._llcrypt registered [ 1024.101475] Key type .llcrypt registered [ 1025.147874] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:24:01 (1786710241) [ 1026.995726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1028.838585] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:24:04 (1786710244) [ 1029.334470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.365397] alg: No test for adler32 (adler32-zlib) [ 1030.263538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.309745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1030.343294] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1031.063779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.52 [ 1031.080499] LNet: 54184:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1031.086820] LNet: Removed LNI 192.168.203.52@tcp [ 1037.501303] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:24:13 (1786710253) [ 1037.600191] LNet: 54746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.607390] LNetError: 54746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.632924] LNet: Removed LNI 192.168.203.52@tcp [ 1038.512572] Key type .llcrypt unregistered [ 1038.514930] Key type ._llcrypt unregistered [ 1044.689459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1044.905543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1047.630794] Key type ._llcrypt registered [ 1047.634414] Key type .llcrypt registered [ 1047.780433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1047.797059] alg: No test for adler32 (adler32-zlib) [ 1049.119720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.084462] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:24:31 (1786710271) [ 1056.268753] LNet: 55663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.283625] LNetError: 55663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.298081] LNet: Removed LNI 10.1.2.3@tcp [ 1057.094649] Key type .llcrypt unregistered [ 1057.097540] Key type ._llcrypt unregistered [ 1064.108734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.347135] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.357107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1064.371969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.173956] Key type ._llcrypt registered [ 1067.183814] Key type .llcrypt registered [ 1067.408640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.425959] alg: No test for adler32 (adler32-zlib) [ 1068.532824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.451602] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:24:51 (1786710291) [ 1078.376692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1080.467794] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:24:56 (1786710296) [ 1080.979809] LNet: 56864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.992199] LNetError: 56864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1081.016095] LNet: Removed LNI 10.1.2.3@tcp [ 1081.767477] Key type .llcrypt unregistered [ 1081.770961] Key type ._llcrypt unregistered [ 1089.398063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.652871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.658811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.667158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1092.292684] Key type ._llcrypt registered [ 1092.295355] Key type .llcrypt registered [ 1092.499529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.517266] alg: No test for adler32 (adler32-zlib) [ 1093.515033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.618709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1103.487241] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:25:19 (1786710319) [ 1103.934706] LNet: 58098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.942573] LNetError: 58098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1103.963341] LNet: Removed LNI 10.1.2.3@tcp [ 1104.747857] Key type .llcrypt unregistered [ 1104.755135] Key type ._llcrypt unregistered [ 1111.603618] Key type ._llcrypt registered [ 1111.608365] Key type .llcrypt registered [ 1112.021284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.044693] alg: No test for adler32 (adler32-zlib) [ 1112.903645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.930665] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1121.401596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1122.353106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_interrupt -r 1 [ 1122.404388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_interrupt -r 1 [ 1122.456918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1122.489630] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1122.746892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.799294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.845230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.889717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.966261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.008486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.062272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.116149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.129964] LNet: 58541:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.164956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.175170] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1123.180647] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1123.240431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.297198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.339774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.386500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.435314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.482923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.060699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_dropped -r 1 [ 1124.088849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_dropped -r 1 [ 1124.134849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1124.384285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.413490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.443285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.469708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.513326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.544329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.578766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.625644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.643144] LNet: 58650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.666362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.692760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.725416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.750652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.774712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.798501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.828095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.305956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_aborted -r 1 [ 1125.335793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_aborted -r 1 [ 1125.364720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1125.381442] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1125.385698] LNet: Skipped 1 previous similar message [ 1125.618981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.645870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.679421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.721070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.762410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.805204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.834599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.865966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.892897] LNet: 58760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.913750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.957418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.994563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.035749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.060646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.092370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.125379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.613466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_no_route -r 1 [ 1126.648896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_no_route -r 1 [ 1126.686340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1126.954202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.994081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.040955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.076147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.111870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.168801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.211163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.246407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.282733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.345583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.387991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.427449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.469581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.502828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.537819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.066547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_timeout -r 1 [ 1128.120577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_timeout -r 1 [ 1128.165913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1128.185170] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1128.191573] LNet: Skipped 1 previous similar message [ 1128.450850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.488660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.530986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.566449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.589829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.615898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.644187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.668570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.693771] LNet: 58978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.705766] LNet: 58978:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1128.729652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.771507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.803692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.839769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.872124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.895638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.928432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1129.368368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_error -r 1 [ 1129.402110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_error -r 1 [ 1129.436625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1129.709501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.744406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.772485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.804258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.833859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.861235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.893432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.924226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.951354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.970954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.998258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.030651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.071406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.119644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.157908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.415191] LNet: 59133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.424780] LNetError: 59133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.442019] LNet: Removed LNI 192.168.203.52@tcp [ 1130.798868] Key type .llcrypt unregistered [ 1130.805263] Key type ._llcrypt unregistered [ 1148.704452] Key type ._llcrypt registered [ 1148.706072] Key type .llcrypt registered [ 1150.060077] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:26:05 (1786710365) [ 1151.469432] Key type .llcrypt unregistered [ 1151.485977] Key type ._llcrypt unregistered [ 1157.942790] Key type ._llcrypt registered [ 1157.949470] Key type .llcrypt registered [ 1158.405508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.447175] alg: No test for adler32 (adler32-zlib) [ 1159.456547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.538820] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1168.749570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1183.127047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1183.205599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.246683] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1185.401176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_interrupt -r 1 [ 1185.436668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_interrupt -r 1 [ 1185.501520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_interrupt -r 1 [ 1185.565512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_interrupt -r 1 [ 1185.626776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_interrupt -r 1 [ 1185.730557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_interrupt -r 1 [ 1185.804772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_interrupt -r 1 [ 1185.862643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_interrupt -r 1 [ 1185.953585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1185.992711] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1186.002295] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1186.274941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.317729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.358690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.386366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.426374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.535071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.602776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.636943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.650965] LNet: 60620:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.663448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.684282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.707510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.745283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.784388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.824681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.886739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.627681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_dropped -r 1 [ 1187.681750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_dropped -r 1 [ 1187.743603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_dropped -r 1 [ 1187.806378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_dropped -r 1 [ 1187.888995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_dropped -r 1 [ 1187.935803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_dropped -r 1 [ 1187.987179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_dropped -r 1 [ 1188.021027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_dropped -r 1 [ 1188.055479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1188.086929] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1188.094056] LNet: Skipped 3 previous similar messages [ 1188.415781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.458304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.523913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.590985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.656316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.741837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.795924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.858150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.895688] LNet: 60742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.925388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.991921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.061868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.127570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.154450] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1189.164667] LNet: Skipped 2 previous similar messages [ 1189.214093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.262691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.299709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.304810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_aborted -r 1 [ 1190.427688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_aborted -r 1 [ 1190.519288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_aborted -r 1 [ 1190.626397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_aborted -r 1 [ 1190.715122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_aborted -r 1 [ 1190.799911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_aborted -r 1 [ 1190.895632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_aborted -r 1 [ 1190.977623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_aborted -r 1 [ 1191.058137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1191.333508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.389592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.456600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.516263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.553801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.616541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.687192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.743258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.781681] LNet: 60863:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.805456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.870254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.921561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.958508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.021735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.070410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.124562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1192.849385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_no_route -r 1 [ 1192.896538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_no_route -r 1 [ 1192.959431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_no_route -r 1 [ 1193.016346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_no_route -r 1 [ 1193.063924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_no_route -r 1 [ 1193.121982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_no_route -r 1 [ 1193.179510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_no_route -r 1 [ 1193.212696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_no_route -r 1 [ 1193.239234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1193.256358] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1193.262431] LNet: Skipped 3 previous similar messages [ 1193.478767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.508412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.537571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.564553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.589316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.611239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.639400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.668483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.707595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.747314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.772949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.794949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.825257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.857213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.890435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.283877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_timeout -r 1 [ 1194.335137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_timeout -r 1 [ 1194.380765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_timeout -r 1 [ 1194.430460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_timeout -r 1 [ 1194.479374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_timeout -r 1 [ 1194.517820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_timeout -r 1 [ 1194.548116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_timeout -r 1 [ 1194.581742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_timeout -r 1 [ 1194.636598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1194.935447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.999611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.064553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.105727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.148443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.195700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.242718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.298310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.340458] LNet: 61105:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.362591] LNet: 61105:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1195.429755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.514930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.576768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.599350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.628896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.665988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.715717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.365230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_error -r 1 [ 1196.421295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_error -r 1 [ 1196.492938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_error -r 1 [ 1196.561443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_error -r 1 [ 1196.611874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_error -r 1 [ 1196.669578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_error -r 1 [ 1196.732184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_error -r 1 [ 1196.786378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_error -r 1 [ 1196.855178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1197.170461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.221174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.274237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.306455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.348888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.393533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.459417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.519809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.594824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.675903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.705279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.731916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.758844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.808313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.845823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.924528] LNet: 61307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.932849] LNetError: 61307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.945373] LNet: Removed LNI 192.168.203.52@tcp [ 1204.971903] LNet: Removed LNI 192.168.203.52@tcp1 [ 1205.531690] Key type .llcrypt unregistered [ 1205.535865] Key type ._llcrypt unregistered [ 1223.146517] Key type ._llcrypt registered [ 1223.148447] Key type .llcrypt registered [ 1224.646579] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:27:20 (1786710440) [ 1225.932263] Key type .llcrypt unregistered [ 1225.942403] Key type ._llcrypt unregistered [ 1232.655469] Key type ._llcrypt registered [ 1232.662359] Key type .llcrypt registered [ 1232.912476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1232.926941] alg: No test for adler32 (adler32-zlib) [ 1233.882674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1233.908147] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1242.042200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1242.664834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_dropped -r 1 [ 1242.700819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_dropped -r 1 [ 1242.742258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1242.760485] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1242.960099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.999938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.023472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.054869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.089584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.120658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.147042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.187523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.219664] LNet: 62562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.237265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.273400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.303834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.336505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.379938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.410507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.453179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.975539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_error -r 1 [ 1244.013969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_error -r 1 [ 1244.046208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1244.065910] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1244.279683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.322081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.368321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.401303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.451206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.483280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.514616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.565461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.593226] LNet: 62671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.617199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.713328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.760236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.825353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.899720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.949441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.999376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.477672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_timeout -r 1 [ 1245.541984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_timeout -r 1 [ 1245.616666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1245.647727] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1245.881400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.910681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.944303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.979667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.037260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.089625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.151771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.191579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.221517] LNet: 62780:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.241120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.289993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.314880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.341484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.370776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.398488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.430654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.665960] LNet: 62825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.673727] LNetError: 62825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.692808] LNet: Removed LNI 192.168.203.52@tcp [ 1247.160452] Key type .llcrypt unregistered [ 1247.164658] Key type ._llcrypt unregistered [ 1264.604802] Key type ._llcrypt registered [ 1264.607781] Key type .llcrypt registered [ 1265.732194] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:28:01 (1786710481) [ 1266.581406] Key type .llcrypt unregistered [ 1266.585332] Key type ._llcrypt unregistered [ 1273.772258] Key type ._llcrypt registered [ 1273.775989] Key type .llcrypt registered [ 1274.072380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.112669] alg: No test for adler32 (adler32-zlib) [ 1275.069775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1275.099741] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1283.665610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1293.170483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.233197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1293.263708] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1294.689284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_dropped -r 1 [ 1294.747628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e remote_dropped -r 1 [ 1294.807919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_dropped -r 1 [ 1294.873073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e remote_dropped -r 1 [ 1294.938148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e remote_dropped -r 1 [ 1295.009838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e remote_dropped -r 1 [ 1295.054827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e remote_dropped -r 1 [ 1295.095481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e remote_dropped -r 1 [ 1295.152518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1295.173582] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1295.181774] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.152@tcp [ 1295.416774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.443946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.473424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.508281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.535858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.580554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.628644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.689743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.714921] LNet: 64298:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.739700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.774106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.815482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.853973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.905904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.950781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.989407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.044230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.683422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_error -r 1 [ 1296.745033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e remote_error -r 1 [ 1296.791640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_error -r 1 [ 1296.849805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e remote_error -r 1 [ 1296.893465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e remote_error -r 1 [ 1296.951988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e remote_error -r 1 [ 1296.992723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e remote_error -r 1 [ 1297.040424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e remote_error -r 1 [ 1297.086988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1297.102848] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1297.111982] LNet: Skipped 2 previous similar messages [ 1297.356139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.395517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.443115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.499249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.552846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.619570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.655849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.679837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.694677] LNet: 64421:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.717546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.748888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.783413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.816376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.851734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.904328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.943371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1297.993588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.562895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_timeout -r 1 [ 1298.611221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e remote_timeout -r 1 [ 1298.657852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_timeout -r 1 [ 1298.706169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e remote_timeout -r 1 [ 1298.753263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e remote_timeout -r 1 [ 1298.812361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e remote_timeout -r 1 [ 1298.881729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e remote_timeout -r 1 [ 1298.921188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e remote_timeout -r 1 [ 1298.968569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.152@tcp [ 1298.986749] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1299.249360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.286614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1299.319421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.345404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.372640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.407560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.450825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.491069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.504602] LNet: 64545:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.525400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.557552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.592763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.617576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.673334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.752871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.798021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.851958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.289801] LNet: 64627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.300592] LNetError: 64627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.318206] LNet: Removed LNI 192.168.203.52@tcp [ 1305.341836] LNet: Removed LNI 192.168.203.52@tcp1 [ 1305.732796] Key type .llcrypt unregistered [ 1305.742390] Key type ._llcrypt unregistered [ 1322.359256] Key type ._llcrypt registered [ 1322.367271] Key type .llcrypt registered [ 1323.896424] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:28:59 (1786710539) [ 1325.665266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1327.510467] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:29:03 (1786710543) [ 1328.547504] Key type .llcrypt unregistered [ 1328.551366] Key type ._llcrypt unregistered [ 1334.627505] Key type ._llcrypt registered [ 1334.628600] Key type .llcrypt registered [ 1334.766482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1334.778807] alg: No test for adler32 (adler32-zlib) [ 1335.742542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1335.773852] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1345.039227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1346.238273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e network_timeout -r 1 [ 1346.368781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e network_timeout -r 1 [ 1346.481825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1346.526970] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1346.862788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1346.916990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1346.961387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.014438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.039391] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1347.060611] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1347.122626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.185383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.236801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.307394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.350525] LNet: 66178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.389627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.477578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.555120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.659385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.704286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.770377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.180487] LNet: 66221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.190813] LNetError: 66221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.226841] LNet: Removed LNI 192.168.203.52@tcp [ 1349.697927] Key type .llcrypt unregistered [ 1349.700737] Key type ._llcrypt unregistered [ 1371.470839] Key type ._llcrypt registered [ 1371.473388] Key type .llcrypt registered [ 1371.791844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1371.822871] alg: No test for adler32 (adler32-zlib) [ 1372.758650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1372.781708] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1380.538508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1390.558311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.655836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.687806] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1392.470741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e network_timeout -r 1 [ 1392.522790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e network_timeout -r 1 [ 1392.572477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e network_timeout -r 1 [ 1392.654712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e network_timeout -r 1 [ 1392.698454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e network_timeout -r 1 [ 1392.747413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e network_timeout -r 1 [ 1392.793443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e network_timeout -r 1 [ 1392.861888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e network_timeout -r 1 [ 1392.935615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1392.962047] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1393.294344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1393.313853] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1393.323963] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.152@tcp [ 1393.326862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1393.356850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1393.386937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1393.422530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1393.454637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1393.497140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1393.582587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1393.619774] LNet: 67107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1393.637208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1393.671493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1393.735532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1393.772542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1393.843244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1393.928036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.469672] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1399.485140] LNet: Skipped 2 previous similar messages [ 1399.952631] LNet: 67185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.967491] LNetError: 67185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.985133] LNet: Removed LNI 192.168.203.52@tcp [ 1400.000337] LNet: Removed LNI 192.168.203.52@tcp1 [ 1400.496812] Key type .llcrypt unregistered [ 1400.499829] Key type ._llcrypt unregistered [ 1419.829800] Key type ._llcrypt registered [ 1419.832315] Key type .llcrypt registered [ 1421.109637] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:30:36 (1786710636) [ 1421.507655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1421.521810] alg: No test for adler32 (adler32-zlib) [ 1422.478601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.546458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.565824] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1423.458520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.484915] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1424.173051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 1424.260054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1424.306100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1424.367952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1424.439970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1424.501683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1424.555332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.52@tcp [ 1424.584563] LNetError: 68286:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.52@tcp to recovery [ 1425.567303] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1425.567724] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1427.615696] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1428.152511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1431.712609] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1432.765884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1432.797928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1432.867882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1432.905613] LNet: 68404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.918395] LNetError: 68404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.951183] LNet: Removed LNI 192.168.203.52@tcp [ 1433.956643] LNetError: 68404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.992566] LNet: Removed LNI 192.168.203.52@tcp1 [ 1434.042935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.135730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.190987] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1435.095759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1435.139260] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1436.273390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 1436.376472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1436.446488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1436.535582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1436.594267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1436.642622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1436.676140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1436.707186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.52@tcp [ 1436.736895] LNetError: 68676:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.52@tcp to recovery [ 1437.152970] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1440.286792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1443.295621] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1443.302125] LNet: Skipped 1 previous similar message [ 1448.282633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1448.328986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1448.359889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1454.995143] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:31:11 (1786710671) [ 1455.374387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.392859] LNet: 69395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.401987] LNetError: 69395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.409618] LNet: Removed LNI 192.168.203.52@tcp [ 1455.421125] LNet: Removed LNI 192.168.203.52@tcp1 [ 1455.450869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.514658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.561555] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1456.106576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.126832] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1456.897147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 1456.954636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1456.982103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1457.010797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1457.032970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1457.051488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.071795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.52@tcp --health 0 [ 1457.157319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.567180] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.52@tcp [ 1458.591457] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1458.602276] LNet: Skipped 1 previous similar message [ 1473.795723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.018169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1474.055194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1474.104912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.52@tcp --health 500 [ 1474.198364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.898625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.970469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.996662] LNet: 69895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.005766] LNetError: 69895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.014444] LNetError: 69895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.024875] LNet: Removed LNI 192.168.203.52@tcp [ 1481.044456] LNet: Removed LNI 192.168.203.52@tcp1 [ 1481.070930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.143966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.181434] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1481.850297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.865144] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1482.671274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 1482.711900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1482.771535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1482.820385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1482.849265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1482.893594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1482.935842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.983671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.52@tcp --health 0 [ 1484.191449] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1484.203076] LNet: Skipped 3 previous similar messages [ 1485.364211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.459142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1495.495771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1495.531683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1502.672401] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:31:58 (1786710718) [ 1503.125416] LNet: 70952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.129933] LNetError: 70952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.132289] LNetError: 70952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1503.137768] LNet: Removed LNI 192.168.203.52@tcp [ 1503.147331] LNet: Removed LNI 192.168.203.52@tcp1 [ 1503.731792] Key type .llcrypt unregistered [ 1503.733933] Key type ._llcrypt unregistered [ 1509.122599] Key type ._llcrypt registered [ 1509.125190] Key type .llcrypt registered [ 1509.320585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1509.339465] alg: No test for adler32 (adler32-zlib) [ 1510.292696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1510.319989] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1518.567892] Lustre: DEBUG MARKER: Initial discovery [ 1518.701106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.152@tcp [ 1520.946505] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1521.161566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1521.225414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.152@tcp [ 1521.268892] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1521.302245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1521.339335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.152@tcp -r 1 -m GET -l 3 [ 1521.396787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.152@tcp -r 1 -m GET -e local_error [ 1521.465154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.152@tcp -r 1 -m PUT -l 6 [ 1521.567127] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1523.086202] Lustre: DEBUG MARKER: Force oleg352-client.virtnet to discover 192.168.203.152@tcp (in background) [ 1523.429671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.152@tcp [ 1524.662692] Lustre: DEBUG MARKER: Force oleg352-server to discover 192.168.203.52@tcp [ 1526.496142] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1527.134681] Lustre: DEBUG MARKER: Wait for 71720 [ 1528.918284] Lustre: DEBUG MARKER: Finished wait on 71720 [ 1529.141822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1529.208781] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1529.217989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1529.451722] LNet: 72175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.457731] LNetError: 72175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.480928] LNet: Removed LNI 192.168.203.52@tcp [ 1529.897446] Key type .llcrypt unregistered [ 1529.905292] Key type ._llcrypt unregistered [ 1548.484071] Key type ._llcrypt registered [ 1548.488045] Key type .llcrypt registered [ 1549.764845] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:32:45 (1786710765) [ 1550.469775] Key type .llcrypt unregistered [ 1550.471756] Key type ._llcrypt unregistered [ 1558.270251] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1558.464440] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1558.492640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1558.502840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1561.254266] Key type ._llcrypt registered [ 1561.255841] Key type .llcrypt registered [ 1561.481944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1561.498219] alg: No test for adler32 (adler32-zlib) [ 1562.434336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.502793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.535028] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1563.250505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.270929] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1574.877148] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:33:10 (1786710790) [ 1575.371914] LNet: 74259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.381241] LNetError: 74259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1575.388761] LNet: Removed LNI 10.1.2.3@tcp [ 1575.398875] LNet: Removed LNI 192.168.203.52@tcp [ 1575.983975] Key type .llcrypt unregistered [ 1575.991342] Key type ._llcrypt unregistered [ 1583.789675] Key type ._llcrypt registered [ 1583.791377] Key type .llcrypt registered [ 1584.101819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1584.129341] alg: No test for adler32 (adler32-zlib) [ 1585.216466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.356916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.397471] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1586.335262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1586.351734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1587.330220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.52@tcp [ 1588.398895] LNetError: Unexpected error -22 connecting to 192.168.203.52@tcp at host 192.168.203.52:988 [ 1595.096210] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:33:30 (1786710810) [ 1595.241954] LNet: 75387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.253106] LNetError: 75387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1595.259693] LNet: Removed LNI 10.1.2.3@tcp [ 1595.278369] LNet: Removed LNI 192.168.203.52@tcp [ 1595.961415] Key type .llcrypt unregistered [ 1595.965561] Key type ._llcrypt unregistered [ 1602.226106] Key type ._llcrypt registered [ 1602.233875] Key type .llcrypt registered [ 1602.397422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1602.409730] alg: No test for adler32 (adler32-zlib) [ 1603.267702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.326684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1603.354245] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1604.104869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1604.152400] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 1605.165315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.52@tcp1 --nid 192.168.203.52@tcp2 [ 1609.597938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp1 192.168.203.52@tcp2 [ 1609.618374] LNetError: 76236:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.52@tcp1 to 192.168.203.52@tcp2 - Route table may be misconfigured [ 1609.645309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp2 192.168.203.52@tcp1 [ 1616.815275] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:33:52 (1786710832) [ 1616.933275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.970989] LNet: 76805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.977297] LNetError: 76805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.990887] LNet: Removed LNI 192.168.203.52@tcp1 [ 1618.026251] LNet: Removed LNI 192.168.203.52@tcp2 [ 1618.074777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.172766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.224597] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1618.904477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.924508] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1619.794781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 1619.844672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.52@tcp [ 1619.881143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp1 192.168.203.52@tcp1 [ 1619.943166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.52@tcp -d 192.168.203.52@tcp -e network_timeout [ 1620.008459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -e network_timeout [ 1620.066615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -e network_timeout [ 1620.110690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -e network_timeout [ 1620.145581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1620.194789] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1620.218982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1620.349560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1620.440839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1621.226209] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1627.138948] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:34:03 (1786710843) [ 1627.562570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.600797] LNet: 77673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.607388] LNetError: 77673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.615300] LNetError: 77673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1627.622090] LNet: Removed LNI 192.168.203.52@tcp [ 1628.647626] LNet: Removed LNI 192.168.203.52@tcp1 [ 1628.690446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.793784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1629.102134] LNet: 77714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.563238] Key type .llcrypt unregistered [ 1629.567089] Key type ._llcrypt unregistered [ 1641.851894] Key type ._llcrypt registered [ 1641.856256] Key type .llcrypt registered [ 1643.990364] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:34:18 (1786710858) [ 1644.466728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1644.483686] alg: No test for adler32 (adler32-zlib) [ 1645.451975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.037834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1649.154920] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:34:24 (1786710864) [ 1649.346174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.378847] LNet: 78810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.433366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.565585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.600681] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1650.721347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.751305] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1651.889879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1651.910623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp1 [ 1651.938940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp1 [ 1658.557450] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:34:34 (1786710874) [ 1660.172051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg352-server [ 1661.858630] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:34:37 (1786710877) [ 1663.712365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg352-server [ 1665.492089] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:34:41 (1786710881) [ 1667.030324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg352-server [ 1669.017993] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:34:44 (1786710884) [ 1670.683430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg352-server [ 1672.679231] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:34:48 (1786710888) [ 1674.302644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg352-server [ 1675.840339] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:34:51 (1786710891) [ 1677.892295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg352-server [ 1679.801285] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:34:55 (1786710895) [ 1681.999451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg352-server [ 1683.590217] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:34:59 (1786710899) [ 1685.202859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg352-server [ 1686.829175] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:35:02 (1786710902) [ 1688.642336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg352-server [ 1690.281250] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:35:06 (1786710906) [ 1690.575834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.615504] LNet: 82690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.635368] LNetError: 82690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.646560] LNet: Removed LNI 192.168.203.52@tcp [ 1691.695812] LNet: Removed LNI 192.168.203.52@tcp1 [ 1691.742893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.834359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.866550] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1692.590638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1692.672714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1692.862735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.900878] LNet: 82838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.916771] LNetError: 82838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.923032] LNetError: 82838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1692.952081] LNet: Removed LNI 192.168.203.52@tcp [ 1692.986515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.061893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1693.095217] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1693.841137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1694.060799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.086942] LNetError: 82983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.144442] LNet: Removed LNI 192.168.203.52@tcp [ 1695.173697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.265574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.290853] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1695.934569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1696.045698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1696.308723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.324513] LNet: 83130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.329982] LNet: 83130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1696.337172] LNetError: 83130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.365102] LNet: Removed LNI 192.168.203.52@tcp [ 1696.389768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.452487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1696.486250] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1697.349332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1697.576769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.664378] LNet: Removed LNI 192.168.203.52@tcp [ 1698.698878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.786037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.812928] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1699.517827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1699.604699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1700.981294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.011670] LNet: 83432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.026063] LNet: 83432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1701.044614] LNetError: 83432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.048137] LNetError: 83432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1701.100661] LNet: Removed LNI 192.168.203.52@tcp [ 1701.157720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.245222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1701.280811] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1702.035454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1702.312474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.399519] LNet: Removed LNI 192.168.203.52@tcp [ 1703.425829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.531221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.586518] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1704.461723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1704.546851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1705.959035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.055923] LNet: Removed LNI 192.168.203.52@tcp [ 1707.092494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.149708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1707.176686] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1707.921816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1708.194518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.292599] LNet: Removed LNI 192.168.203.52@tcp [ 1709.322696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.412638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.470654] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1710.316865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1710.413634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1710.668421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.686350] LNet: 84027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.695423] LNet: 84027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1710.705981] LNetError: 84027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.711151] LNetError: 84027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1711.782456] LNet: Removed LNI 192.168.203.52@tcp [ 1711.833937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.928967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1711.971236] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1712.901828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1713.241902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.280736] LNet: Removed LNI 192.168.203.52@tcp [ 1714.310065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.399730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.431758] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1715.183171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1715.279558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1715.610406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.719031] LNet: Removed LNI 192.168.203.52@tcp [ 1716.768859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.853726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1716.877790] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1717.694519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 1718.033246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.142771] LNet: Removed LNI 192.168.203.52@tcp [ 1719.181250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.239475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.271445] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1720.003564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1720.071064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.106560] LNet: Removed LNI 192.168.203.52@tcp [ 1720.140529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.202254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.241209] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1720.887553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1720.963770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.992867] LNet: Removed LNI 192.168.203.52@tcp [ 1721.019536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.082655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1721.188859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.266150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.341960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1721.447121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.516436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.589267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1721.680737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.721436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.782309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1721.813792] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1722.553576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.584966] LNet: Removed LNI 192.168.203.52@tcp [ 1722.611756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.696158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1729.905634] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:35:45 (1786710945) [ 1730.138312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.161830] LNet: 85487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.167705] LNet: 85487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1730.191737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.258659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1730.294022] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/180] [ 1731.797269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.822790] LNetError: 85625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.829154] LNetError: 85625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1731.843703] LNet: Removed LNI 192.168.203.52@tcp231 [ 1731.872175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.945248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1731.979267] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/180] [ 1731.989936] LNet: 85633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1732.607910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1732.659782] LNet: Removed LNI 192.168.203.52@tcp231 [ 1732.675226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1732.706935] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/180] [ 1734.051647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.080928] LNet: Removed LNI 192.168.203.52@tcp231 [ 1734.108059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.191524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1734.228370] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/0] [ 1734.232763] LNet: 85785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1734.740944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1734.784492] LNet: Removed LNI 192.168.203.52@tcp231 [ 1734.801474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1734.830751] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/0] [ 1735.872594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.905846] LNet: Removed LNI 192.168.203.52@tcp231 [ 1735.960786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.069181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1736.103398] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/60] [ 1736.107775] LNet: 85937:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1736.625282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1736.662418] LNet: Removed LNI 192.168.203.52@tcp231 [ 1736.681152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1736.722233] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/60] [ 1737.983023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.046812] LNet: Removed LNI 192.168.203.52@tcp231 [ 1738.074490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.205472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1738.248843] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/180] [ 1738.260758] LNet: 86088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1738.741260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1738.788350] LNet: Removed LNI 192.168.203.52@tcp231 [ 1738.811812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1738.837034] LNet: Added LNI 192.168.203.52@tcp231 [8/256/0/180] [ 1749.081250] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:36:04 (1786710964) [ 1749.238662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.313076] LNet: Removed LNI 192.168.203.52@tcp231 [ 1749.376045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.567085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1749.598629] LNet: Added LNI 192.168.203.52@tcp232 [8/256/0/180] [ 1750.867631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1750.920933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1750.960152] LNet: Removed LNI 192.168.203.52@tcp232 [ 1750.987288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1751.022839] LNet: Added LNI 192.168.203.52@tcp232 [8/256/0/180] [ 1752.081577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1752.133200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1752.191139] LNet: Removed LNI 192.168.203.52@tcp232 [ 1752.216340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1752.254160] LNet: Added LNI 192.168.203.52@tcp232 [8/256/0/180] [ 1753.151957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1753.194873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1753.232349] LNet: Removed LNI 192.168.203.52@tcp232 [ 1753.263500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1753.305149] LNet: Added LNI 192.168.203.52@tcp232 [8/256/0/180] [ 1754.067370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1754.113847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1754.151638] LNet: Removed LNI 192.168.203.52@tcp232 [ 1754.179097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1761.547805] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:36:17 (1786710977) [ 1762.971542] Key type .llcrypt unregistered [ 1762.977758] Key type ._llcrypt unregistered [ 1769.914571] Key type ._llcrypt registered [ 1769.916884] Key type .llcrypt registered [ 1770.121423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1770.138259] alg: No test for adler32 (adler32-zlib) [ 1771.087509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1771.115637] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1778.231580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1787.848846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.900431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1787.919966] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1789.516862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_interrupt -r 2 [ 1789.563456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_interrupt -r 2 [ 1789.592646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_interrupt -r 2 [ 1789.636798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_interrupt -r 2 [ 1789.677082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_interrupt -r 2 [ 1789.727129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_interrupt -r 2 [ 1789.794215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_interrupt -r 2 [ 1789.846926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_interrupt -r 2 [ 1789.914897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1789.946677] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1789.967728] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1790.280911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.318795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.351652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.386422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.410692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.443394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.478474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.523589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.549413] LNet: 88599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.569336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.609075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.643713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.673011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.710808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.738183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.191426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_dropped -r 2 [ 1791.225465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_dropped -r 2 [ 1791.269026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_dropped -r 2 [ 1791.312264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_dropped -r 2 [ 1791.394478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_dropped -r 2 [ 1791.456679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_dropped -r 2 [ 1791.506335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_dropped -r 2 [ 1791.566793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_dropped -r 2 [ 1791.641351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1791.664918] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1791.926306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.963851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1792.002944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.049440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.095767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.138170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.179663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.207139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.222555] LNet: 88719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.243894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.268410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.322223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.359159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.383838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.416820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.958548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_aborted -r 2 [ 1792.996973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_aborted -r 2 [ 1793.051550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_aborted -r 2 [ 1793.106992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_aborted -r 2 [ 1793.153421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_aborted -r 2 [ 1793.213175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_aborted -r 2 [ 1793.270503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_aborted -r 2 [ 1793.322683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_aborted -r 2 [ 1793.379619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1793.406273] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1793.638079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1793.660882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1793.687836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1793.724892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1793.764645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1793.804301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1793.846645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1793.903888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1793.922447] LNet: 88839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1793.936282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1793.973990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1794.001159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1794.029463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1794.070393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1794.128544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1794.665744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_no_route -r 2 [ 1794.713947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_no_route -r 2 [ 1794.765943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_no_route -r 2 [ 1794.833542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_no_route -r 2 [ 1794.872776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_no_route -r 2 [ 1794.911191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_no_route -r 2 [ 1794.960575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_no_route -r 2 [ 1794.998239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_no_route -r 2 [ 1795.038509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1795.320462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1795.344447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1795.369048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1795.390910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1795.428184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1795.483668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1795.524289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1795.558334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1795.628850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1795.669085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1795.717333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1795.788648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1795.852833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1795.910987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1796.453315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e local_timeout -r 2 [ 1796.497813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e local_timeout -r 2 [ 1796.546900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e local_timeout -r 2 [ 1796.579285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e local_timeout -r 2 [ 1796.612767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e local_timeout -r 2 [ 1796.652917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e local_timeout -r 2 [ 1796.683022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e local_timeout -r 2 [ 1796.716771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e local_timeout -r 2 [ 1796.763986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1796.814219] LNet: There was an unexpected network error while writing to 192.168.203.152: rc = -22 [ 1796.823361] LNet: Skipped 1 previous similar message [ 1797.075461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1797.117267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1797.151240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1797.186659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1797.220298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1797.262875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1797.302297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1797.334812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1797.364796] LNet: 89080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1797.370173] LNet: 89080:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1797.392766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1797.449897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1797.509548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1797.570994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1797.627516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1797.670771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1798.175495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp -m GET -e remote_dropped -r 2 [ 1798.223402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.152@tcp1 -m GET -e remote_dropped -r 2 [ 1798.278350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp -m GET -e remote_dropped -r 2 [ 1798.323216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp -d 192.168.203.52@tcp1 -m GET -e remote_dropped -r 2 [ 1798.370229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp -m GET -e remote_dropped -r 2 [ 1798.393801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.52@tcp1 -d 192.168.203.152@tcp1 -m GET -e remote_dropped -r 2 [ 1798.428403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp -m GET -e remote_dropped -r 2 [ 1798.462838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.52@tcp1 -d 192.168.203.52@tcp1 -m GET -e remote_dropped -r 2 [ 1798.498940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.52@tcp 192.168.203.152@tcp [ 1798.524542] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.152@tcp [ 1798.817344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.841722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1798.867848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.890745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.922529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.963646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.998825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1799.045660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1799.087230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1799.119582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1799.143756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1799.183401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1799.224304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1799.267735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.498982] LNet: 89278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.502580] LNetError: 89278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.535541] LNet: Removed LNI 192.168.203.52@tcp [ 1805.537658] LNetError: 89278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.559893] LNet: Removed LNI 192.168.203.52@tcp1 [ 1805.969160] Key type .llcrypt unregistered [ 1805.972740] Key type ._llcrypt unregistered [ 1823.039715] Key type ._llcrypt registered [ 1823.042640] Key type .llcrypt registered [ 1824.072060] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:37:20 (1786711040) [ 1824.604144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.620757] alg: No test for adler32 (adler32-zlib) [ 1825.523093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.579232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.598808] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1826.358106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1826.383254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.52@tcp [ 1826.405517] LNetError: 90257:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.52@tcp to recovery [ 1826.511823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.52@tcp [ 1826.614626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1832.488249] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:37:28 (1786711048) [ 1832.654545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.700382] LNet: 90844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.715623] LNetError: 90844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.751087] LNet: Removed LNI 192.168.203.52@tcp [ 1832.777705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.846404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.869782] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1833.441552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1833.473249] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 1834.414779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.52@tcp [ 1834.441849] LNetError: 91113:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.52@tcp to recovery [ 1834.847669] LNet: 1 local NIs in recovery (showing 1): 192.168.203.52@tcp [ 1834.853709] LNet: There was an unexpected network error while writing to 192.168.203.52: rc = -22 [ 1835.671879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.52@tcp [ 1843.548723] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:37:39 (1786711059) [ 1845.392267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg352-server [ 1847.424888] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:37:43 (1786711063) [ 1849.234439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg352-server [ 1851.262983] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:37:47 (1786711067) [ 1853.215731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg352-server [ 1854.712717] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:37:50 (1786711070) [ 1854.976978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.995089] LNet: 92734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.006364] LNetError: 92734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.013689] LNet: Removed LNI 192.168.203.52@tcp [ 1855.029659] LNet: Removed LNI 192.168.203.52@tcp1 [ 1855.064571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.130752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.160950] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1855.986608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1856.059453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1856.115177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.223771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1856.287657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.353628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1863.100813] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:37:58 (1786711078) [ 1863.515937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.554500] LNet: 93460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.574097] LNetError: 93460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.581753] LNetError: 93460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1863.596744] LNet: Removed LNI 192.168.203.52@tcp [ 1863.640740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.733628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.757399] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1872.304609] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:38:07 (1786711087) [ 1872.409008] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.421984] LNet: 94158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.426242] LNetError: 94158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.439369] LNet: Removed LNI 192.168.203.52@tcp [ 1872.458189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.497966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.519663] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1879.477157] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:38:15 (1786711095) [ 1881.123114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1882.688322] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:38:18 (1786711098) [ 1883.399940] LNet: 95198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.407594] LNetError: 95198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.429444] LNet: Removed LNI 192.168.203.52@tcp [ 1884.341652] Key type .llcrypt unregistered [ 1884.345896] Key type ._llcrypt unregistered [ 1890.776579] Key type ._llcrypt registered [ 1890.781790] Key type .llcrypt registered [ 1891.064131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.083361] alg: No test for adler32 (adler32-zlib) [ 1892.029957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1892.055107] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1899.594185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 1904.970805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.152@tcp [ 1905.317319] LNet: 95624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.321575] LNetError: 95624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.337908] LNet: Removed LNI 192.168.203.52@tcp [ 1905.744733] Key type .llcrypt unregistered [ 1905.747385] Key type ._llcrypt unregistered [ 1916.130239] Key type ._llcrypt registered [ 1916.133698] Key type .llcrypt registered [ 1917.303834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1918.930186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1920.501090] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:38:56 (1786711136) [ 1921.927891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1923.494303] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:38:59 (1786711139) [ 1925.474988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg352-server [ 1927.312348] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:39:02 (1786711142) [ 1928.938957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg352-server [ 1930.860044] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:39:06 (1786711146) [ 1932.011486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.036103] alg: No test for adler32 (adler32-zlib) [ 1933.142815] Lustre: Lustre: Build Version: 2.17.57_1_g92903bd [ 1933.328361] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1935.024822] Key type lgssc registered [ 1936.172367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1956.377870] Key type lgssc unregistered [ 1956.675462] LNet: 98804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.691543] LNetError: 98804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.712036] LNet: Removed LNI 192.168.203.52@tcp [ 1957.448637] Key type .llcrypt unregistered [ 1957.451484] Key type ._llcrypt unregistered [ 1972.215077] Key type ._llcrypt registered [ 1972.219415] Key type .llcrypt registered [ 1973.475887] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:39:49 (1786711189) [ 1973.936686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1973.957482] alg: No test for adler32 (adler32-zlib) [ 1974.864885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.932701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1974.992654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1975.031464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1975.074330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.679745] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:39:57 (1786711197) [ 1981.807312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.846585] LNet: 100213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.872871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.933869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.006216] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1983.069234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.113526] LNet: 100365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.124182] LNetError: 100365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.143548] LNet: Removed LNI 192.168.203.52@tcp [ 1983.171627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.265350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1983.289906] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1983.962666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.000640] LNetError: 100393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.020487] LNet: Removed LNI 192.168.203.52@tcp [ 1984.048976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.127827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1984.165651] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1991.647497] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:40:07 (1786711207) [ 1991.767078] LNet: 100976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.774236] LNet: 100976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1991.783845] LNetError: 100976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.816443] LNet: Removed LNI 192.168.203.52@tcp [ 1992.472288] Key type .llcrypt unregistered [ 1992.478629] Key type ._llcrypt unregistered [ 1998.509583] Key type ._llcrypt registered [ 1998.515780] Key type .llcrypt registered [ 1998.836721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1998.853949] alg: No test for adler32 (adler32-zlib) [ 1999.828559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.921499] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 1999.926976] LNet: *** cfs_fail_loc=e003, val=100*** [ 2000.770329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2000.786423] LNet: *** cfs_fail_loc=e003, val=100*** [ 2000.788698] LNet: Skipped 2 previous similar messages [ 2000.795047] Ping reply received with mapping: [ 2000.797463] found metadata reply NI[192.168.203.52@tcp] [ 2000.799594] 34 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2000.803775] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2000.806103] 5a 5a 5a 5a 5a 5a 5a 5a [ 2000.891748] LNet: 101385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.899823] LNetError: 101385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.913684] LNet: Removed LNI 192.168.203.52@tcp [ 2001.488856] Key type .llcrypt unregistered [ 2001.491046] Key type ._llcrypt unregistered [ 2011.633076] Key type ._llcrypt registered [ 2011.635830] Key type .llcrypt registered [ 2012.860524] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:40:28 (1786711228) [ 2013.487363] Key type .llcrypt unregistered [ 2013.489228] Key type ._llcrypt unregistered [ 2019.961371] Key type ._llcrypt registered [ 2019.964033] Key type .llcrypt registered [ 2020.195487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.212612] alg: No test for adler32 (adler32-zlib) [ 2021.431306] LNetError: 102444:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2022.014779] Key type .llcrypt unregistered [ 2022.016630] Key type ._llcrypt unregistered [ 2026.971827] Key type ._llcrypt registered [ 2026.974715] Key type .llcrypt registered [ 2028.295989] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:40:44 (1786711244) [ 2028.851332] Key type .llcrypt unregistered [ 2028.853201] Key type ._llcrypt unregistered [ 2036.604891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2036.609316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2039.235863] Key type ._llcrypt registered [ 2039.239481] Key type .llcrypt registered [ 2039.448541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.459258] alg: No test for adler32 (adler32-zlib) [ 2040.435887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.484620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2049.958994] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:41:05 (1786711265) [ 2050.184156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.205333] LNet: 104233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.210166] LNetError: 104233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2050.224830] LNet: Removed LNI 10.1.2.3@tcp [ 2050.259661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.037776] Key type .llcrypt unregistered [ 2051.040913] Key type ._llcrypt unregistered [ 2057.455601] Key type ._llcrypt registered [ 2057.458068] Key type .llcrypt registered [ 2057.610673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2057.627391] alg: No test for adler32 (adler32-zlib) [ 2058.581777] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2058.647497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2058.835485] LNet: 104526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.851697] LNetError: 104526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.881169] LNet: Removed LNI 192.168.203.52@tcp [ 2059.516684] Key type .llcrypt unregistered [ 2059.517978] Key type ._llcrypt unregistered [ 2071.363300] Key type ._llcrypt registered [ 2071.366185] Key type .llcrypt registered [ 2072.603570] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:41:28 (1786711288) [ 2073.544300] Key type .llcrypt unregistered [ 2073.548565] Key type ._llcrypt unregistered [ 2079.530406] Key type ._llcrypt registered [ 2079.533472] Key type .llcrypt registered [ 2079.758645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.775980] alg: No test for adler32 (adler32-zlib) [ 2086.646690] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:41:42 (1786711302) [ 2086.848947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.893837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.930222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.948877] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2087.528822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.583955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2087.630326] LNet: Removed LNI 192.168.203.52@tcp [ 2087.930490] LNet: 106282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.400181] Key type .llcrypt unregistered [ 2088.401996] Key type ._llcrypt unregistered [ 2099.426682] Key type ._llcrypt registered [ 2099.428892] Key type .llcrypt registered [ 2100.762895] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:41:56 (1786711316) [ 2101.138785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.155569] alg: No test for adler32 (adler32-zlib) [ 2102.103619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.174578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.213744] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2102.925503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2102.959174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.52@tcp [ 2102.973849] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2102.977198] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.980165] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 0 [ 2102.988173] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2102.990826] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2102.995471] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.998675] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 0 [ 2103.002830] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2103.005055] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2103.007989] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.011286] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 2152 [ 2103.014056] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2103.016862] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2103.020624] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.024739] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 2152 [ 2103.029733] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2109.702469] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:42:05 (1786711325) [ 2110.218995] LNet: 107823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.223338] LNetError: 107823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.241632] LNet: Removed LNI 192.168.203.52@tcp [ 2110.749382] Key type .llcrypt unregistered [ 2110.751730] Key type ._llcrypt unregistered [ 2118.009311] Key type ._llcrypt registered [ 2118.012952] Key type .llcrypt registered [ 2118.561926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2118.635107] alg: No test for adler32 (adler32-zlib) [ 2119.719828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2119.743852] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2129.939133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 2142.049630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.103367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2142.142302] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 2143.099215] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2143.337499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2143.349528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2143.353313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2145.751299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2145.798360] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2156.970526] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:42:52 (1786711372) [ 2157.218947] LNet: 109338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.228594] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.239520] LNet: Removed LNI 192.168.203.52@tcp [ 2158.303195] LNet: Removed LNI 192.168.203.52@tcp1 [ 2158.307429] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2158.315487] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2158.339750] LNet: Removed LNI 10.1.2.3@tcp99 [ 2159.199641] Key type .llcrypt unregistered [ 2159.203175] Key type ._llcrypt unregistered [ 2167.108279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2167.115562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2169.722260] Key type ._llcrypt registered [ 2169.728882] Key type .llcrypt registered [ 2169.957922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2169.988305] alg: No test for adler32 (adler32-zlib) [ 2170.931830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.985623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.007580] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2171.870246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2171.894284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2173.961169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2174.013809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2174.098483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2174.125647] LNetError: 110088:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2174.174404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2174.236559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2174.781244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2174.870072] LNet: 110109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.874339] LNetError: 110109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2174.883138] LNet: Removed LNI 10.1.2.3@tcp [ 2175.912756] LNet: Removed LNI 192.168.203.52@tcp [ 2176.839055] Key type .llcrypt unregistered [ 2176.849825] Key type ._llcrypt unregistered [ 2190.498128] Key type ._llcrypt registered [ 2190.500832] Key type .llcrypt registered [ 2191.873191] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:43:27 (1786711407) [ 2192.303732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.319703] alg: No test for adler32 (adler32-zlib) [ 2193.209992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.271283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.301302] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2194.063718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2194.166435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg352-client@tcp [ 2200.915497] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:43:36 (1786711416) [ 2201.175774] LNet: 111647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.185937] LNetError: 111647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.207285] LNet: Removed LNI 192.168.203.52@tcp [ 2201.959820] Key type .llcrypt unregistered [ 2201.972221] Key type ._llcrypt unregistered [ 2208.500335] Key type ._llcrypt registered [ 2208.504622] Key type .llcrypt registered [ 2208.729590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.767856] alg: No test for adler32 (adler32-zlib) [ 2209.758680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2209.786980] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2209.820398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2215.934454] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:43:51 (1786711431) [ 2217.183352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2219.089887] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:43:54 (1786711434) [ 2220.481698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2222.368966] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:43:58 (1786711438) [ 2222.500301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.541792] LNet: 113062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.552817] LNetError: 113062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.575733] LNet: Removed LNI 192.168.203.52@tcp [ 2222.608204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.744552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.768349] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2223.738889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2223.780788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2232.352300] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:44:08 (1786711448) [ 2232.596302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.624988] LNet: 113780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.629881] LNetError: 113780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.658899] LNet: Removed LNI 192.168.203.52@tcp [ 2232.680250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.744683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2232.783756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2233.576354] Key type .llcrypt unregistered [ 2233.589266] Key type ._llcrypt unregistered [ 2250.541213] Key type ._llcrypt registered [ 2250.545302] Key type .llcrypt registered [ 2252.127585] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:44:27 (1786711467) [ 2253.246899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.264217] alg: No test for adler32 (adler32-zlib) [ 2254.294073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.396112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.447924] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2255.441756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2255.525188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2261.624073] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:44:37 (1786711477) [ 2261.788887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.811610] LNet: 115339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.825525] LNetError: 115339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.891251] LNet: Removed LNI 192.168.203.52@tcp [ 2262.920177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.003518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2263.053944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2269.068988] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:44:44 (1786711484) [ 2269.314441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.337929] LNet: 115922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.375237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.474244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2269.887185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2269.925427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2270.002301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2270.448854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2270.492928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2270.579762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.597209] LNet: 115955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.625504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.692480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2271.130652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2271.172048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2271.234490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2271.761750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2271.810758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2278.256571] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:44:53 (1786711493) [ 2278.358252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.383186] LNet: 116550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.404151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.518627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2278.548508] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2279.123688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2279.141394] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2280.070734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2280.107961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2280.308441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.349915] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.368151] LNet: Removed LNI 192.168.203.52@tcp [ 2280.385508] LNet: Removed LNI 192.168.203.52@tcp2 [ 2280.424443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.516912] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2280.535520] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2280.546841] LNet: 116841:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2281.081942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2281.298573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.336063] LNet: Removed LNI 192.168.203.52@tcp [ 2282.341550] LNetError: 116878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.352150] LNetError: 116878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2282.371425] LNet: Removed LNI 192.168.203.52@tcp2 [ 2282.403605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.457280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.476248] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2283.081860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2283.097679] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2284.281311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2284.331225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2284.444025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2284.765557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.796460] LNet: 117163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.803716] LNet: 117163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2284.810279] LNetError: 117163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.819096] LNet: Removed LNI 192.168.203.52@tcp [ 2285.862280] LNet: Removed LNI 192.168.203.52@tcp2 [ 2285.915643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.988447] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2285.996668] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2286.002156] LNet: 117169:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.763269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2287.079109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.117250] LNet: Removed LNI 192.168.203.52@tcp [ 2288.171187] LNet: Removed LNI 192.168.203.52@tcp2 [ 2288.212490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.336761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.438220] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2289.336964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2289.380418] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2290.335734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2290.377321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2290.593803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.609031] LNetError: 117485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.616159] LNetError: 117485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2290.625937] LNet: Removed LNI 192.168.203.52@tcp [ 2290.643123] LNet: Removed LNI 192.168.203.52@tcp2 [ 2290.686312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.742945] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2290.747710] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2290.751220] LNet: 117492:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.341451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2291.549718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.639219] LNet: Removed LNI 192.168.203.52@tcp [ 2292.655957] LNet: Removed LNI 192.168.203.52@tcp2 [ 2292.700873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.753347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.767564] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2293.406469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2293.446033] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2294.289743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2294.345449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2294.415149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2294.624877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.648904] LNet: 117813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.655481] LNet: 117813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2295.711273] LNet: Removed LNI 192.168.203.52@tcp [ 2296.757963] LNet: Removed LNI 192.168.203.52@tcp2 [ 2296.793953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.892021] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2296.897813] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2296.901827] LNet: 117819:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.967366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2298.287147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.303475] LNet: Removed LNI 192.168.203.52@tcp [ 2299.366250] LNet: Removed LNI 192.168.203.52@tcp2 [ 2299.404747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.487362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.509868] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2300.300617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2300.326676] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2301.183341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2301.212250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2301.298951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2301.679279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.707485] LNetError: 118153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.712988] LNetError: 118153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2301.720640] LNet: Removed LNI 192.168.203.52@tcp [ 2301.738559] LNet: Removed LNI 192.168.203.52@tcp2 [ 2301.780254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.843162] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2301.846675] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2301.849692] LNet: 118159:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.137986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2302.435742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.520515] LNet: Removed LNI 192.168.203.52@tcp [ 2303.539559] LNet: Removed LNI 192.168.203.52@tcp2 [ 2303.568990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.644491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.682514] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2304.456923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2304.506160] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2305.415499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2305.453401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2305.479616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2305.531835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2305.838743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.911400] LNet: Removed LNI 192.168.203.52@tcp [ 2307.944301] LNet: Removed LNI 192.168.203.52@tcp2 [ 2307.976561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.054248] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2308.065047] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2308.357501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2315.730510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2317.567874] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:45:33 (1786711533) [ 2317.705051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.730366] LNet: 119273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.738022] LNet: 119273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2317.744767] LNetError: 119273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.750621] LNetError: 119273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2317.755875] LNet: Removed LNI 192.168.203.52@tcp [ 2318.861292] LNet: Removed LNI 192.168.203.52@tcp2 [ 2318.919249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.179812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.205800] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2319.949986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2319.962992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2320.941526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.972463] LNet: Removed LNI 10.1.2.3@tcp [ 2320.998639] LNet: Removed LNI 192.168.203.52@tcp [ 2321.012359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.105569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2321.143750] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2321.148083] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2321.662299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2321.686958] LNet: Removed LNI 192.168.203.52@tcp [ 2321.704636] LNet: Removed LNI 10.1.2.3@tcp [ 2328.393707] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:45:44 (1786711544) [ 2328.494114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.559055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.684380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.708367] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2329.659434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2329.740847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2329.799933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2329.870916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2329.909254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2329.978937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.019250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.065729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.115201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.181461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.229604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.52@tcp [ 2330.403326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2338.249583] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:45:53 (1786711553) [ 2338.352106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.439261] LNet: Removed LNI 192.168.203.52@tcp [ 2339.494471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.703074] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2339.785529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2339.792840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2339.801838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2342.314516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.358900] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2343.200314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2343.220162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2345.410108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2345.474877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2345.560095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.52@tcp -m PUT -e local_timeout -r 1 [ 2345.658785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2345.771230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2345.780781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2346.157263] LNetError: 120960:0:(lib-move.c:4801:lnet_parse()) 192.168.203.52@tcp, src 192.168.203.52@tcp: Dropping PUT (error -108 looking up sender) [ 2347.168283] LNet: Removed LNI 10.1.2.3@tcp [ 2348.201906] LNet: Removed LNI 192.168.203.52@tcp [ 2348.714269] Key type .llcrypt unregistered [ 2348.716314] Key type ._llcrypt unregistered [ 2362.765059] Key type ._llcrypt registered [ 2362.771417] Key type .llcrypt registered [ 2364.299199] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:46:19 (1786711579) [ 2364.958155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.000562] alg: No test for adler32 (adler32-zlib) [ 2366.042970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.591795] LNet: 122277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.503713] Key type .llcrypt unregistered [ 2367.505612] Key type ._llcrypt unregistered [ 2374.395154] Key type ._llcrypt registered [ 2374.396919] Key type .llcrypt registered [ 2374.660534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2374.691934] alg: No test for adler32 (adler32-zlib) [ 2375.633257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2375.661350] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2384.081434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 2384.297623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2384.906564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.152@tcp [ 2385.221775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.256796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2385.282894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.315792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.361442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.402058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.439630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.476369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.496540] LNet: 122732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.516601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.553391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2385.589749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2385.623253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2385.658988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2385.696291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.380813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2386.635888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2386.673355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2386.714244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2386.739158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2386.767796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2386.800472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2386.843478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2386.879801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2386.897925] LNet: 122849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2386.911692] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.152@tcp [ 2386.922367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2386.966799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.998870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.046033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.091851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.145911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2387.818593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.152@tcp [ 2388.136509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.182900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2388.221899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.277170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.310914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2388.350572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2388.401302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2388.469670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2388.492146] LNet: 122966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.516794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2388.547084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2388.592359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2388.639156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2388.691763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2388.737920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2389.377109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2389.613258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2389.645925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2389.699533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2389.753042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2389.796767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2389.862379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2389.923188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2389.981381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2390.026214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2390.072514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2390.119759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2390.179286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2390.239195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2390.299112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2390.672617] LNet: 123134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.684314] LNetError: 123134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.702510] LNet: Removed LNI 192.168.203.52@tcp [ 2391.165540] Key type .llcrypt unregistered [ 2391.173534] Key type ._llcrypt unregistered [ 2409.881292] Key type ._llcrypt registered [ 2409.885826] Key type .llcrypt registered [ 2411.494278] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:47:06 (1786711626) [ 2412.995148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.060169] alg: No test for adler32 (adler32-zlib) [ 2414.223991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.936951] LNet: 124022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.191466] Key type .llcrypt unregistered [ 2416.198154] Key type ._llcrypt unregistered [ 2424.591788] Key type ._llcrypt registered [ 2424.598220] Key type .llcrypt registered [ 2424.820123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2424.839353] alg: No test for adler32 (adler32-zlib) [ 2425.786538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2425.813619] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2435.395211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.152@tcp [ 2446.427947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.490473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2446.528673] LNet: Added LNI 192.168.203.52@tcp1 [8/256/0/180] [ 2447.631624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2448.323721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.152@tcp [ 2448.351076] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.152@tcp [ 2448.562267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.607633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.675605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.734347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.789720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.846186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.895113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.932893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.970498] LNet: 124679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2449.000940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.053465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2449.113025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2449.151881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2449.206391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.258575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.141064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2450.426816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.470381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.514552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.558905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.606272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.659932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.718140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2450.751612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2450.771559] LNet: 124796:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2450.789340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2450.813781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.841757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.868907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.897151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.947437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.604818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.152@tcp [ 2451.892308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2451.938539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2451.978825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2452.011108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2452.049430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2452.087487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.118654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2452.143952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2452.161714] LNet: 124914:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.180361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2452.204532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2452.236918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2452.270609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2452.308795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2452.347603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.002695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2453.282967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2453.321563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2453.354335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2453.392263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2453.425132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2453.452698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2453.482811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2453.508698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2453.535961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2453.564807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2453.597778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.630606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.655461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.684695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2458.761844] LNet: 125116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2458.767867] LNetError: 125116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2458.774654] LNet: Removed LNI 192.168.203.52@tcp [ 2458.786888] LNet: Removed LNI 192.168.203.52@tcp1 [ 2459.272635] Key type .llcrypt unregistered [ 2459.275585] Key type ._llcrypt unregistered [ 2476.058262] Key type ._llcrypt registered [ 2476.061095] Key type .llcrypt registered [ 2477.640799] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:48:13 (1786711693) [ 2478.099490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.122462] alg: No test for adler32 (adler32-zlib) [ 2479.064297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.161160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2479.183383] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2480.002514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2480.020713] LNet: Added LNI 192.168.203.52@tcp2 [8/256/0/180] [ 2480.976074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.52@tcp [ 2481.037295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2482.079152] LNet: Removed LNI 192.168.203.52@tcp [ 2488.694558] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:48:24 (1786711704) [ 2490.190934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2491.978369] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:48:27 (1786711707) [ 2498.824980] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:48:34 (1786711714) [ 2504.861230] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:48:41 (1786711721) [ 2510.309885] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:48:46 (1786711726) [ 2516.356959] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:48:52 (1786711732) [ 2522.992441] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:48:58 (1786711738) [ 2529.045431] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:49:05 (1786711745) [ 2534.978819] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:49:10 (1786711750) [ 2541.082647] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:49:17 (1786711757) [ 2547.825730] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:49:23 (1786711763) [ 2554.475650] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:49:30 (1786711770) [ 2562.516218] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:49:38 (1786711778) [ 2570.141076] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:49:45 (1786711785) [ 2578.572266] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:49:53 (1786711793) [ 2587.695468] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:50:03 (1786711803) [ 2596.531640] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:50:11 (1786711811) [ 2605.476025] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:50:20 (1786711820) [ 2616.828900] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:50:32 (1786711832) [ 2617.100374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2617.116447] LNet: 136814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.120582] LNetError: 136814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.150596] LNet: Removed LNI 192.168.203.52@tcp2 [ 2617.189238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2617.366361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2617.387420] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2618.683246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2618.714431] LNet: 136956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2618.727680] LNetError: 136956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.769242] LNet: Removed LNI 192.168.203.52@tcp [ 2618.802979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2618.857905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2618.895258] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2627.254893] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:50:42 (1786711842) [ 2636.082721] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:50:51 (1786711851) [ 2645.059768] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:51:00 (1786711860) [ 2653.771239] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:51:09 (1786711869) [ 2660.854362] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:51:16 (1786711876) [ 2668.323216] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:51:24 (1786711884) [ 2675.125817] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:51:31 (1786711891) [ 2681.710487] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:51:37 (1786711897) [ 2688.276686] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:51:44 (1786711904) [ 2696.142155] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:51:52 (1786711912) [ 2704.357651] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:51:59 (1786711919) [ 2713.554134] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:52:09 (1786711929) [ 2724.361619] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2471 sec ========== 08:52:19 (1786711939) [ 2724.872540] LNet: 144423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.878611] LNetError: 144423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.896259] LNet: Removed LNI 192.168.203.52@tcp [ 2725.735752] Key type .llcrypt unregistered [ 2725.742426] Key type ._llcrypt unregistered [ 2740.883654] Key type ._llcrypt registered [ 2740.885245] Key type .llcrypt registered [ 2741.157829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2741.170539] alg: No test for adler32 (adler32-zlib) [ 2742.150493] Lustre: Lustre: Build Version: 2.17.57_1_g92903bd [ 2742.417400] LNet: Added LNI 192.168.203.52@tcp [8/256/0/180] [ 2744.087640] Key type lgssc registered [ 2745.319774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2810.991588] Lustre: Mounted lustre-client - version 2.17.57_1_g92903bd [ 2815.990759] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2823.625518] Lustre: Unmounted lustre-client [ 2852.229769] Key type lgssc unregistered [ 2852.466982] LNet: 147389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2852.482360] LNetError: 147389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2852.510358] LNet: Removed LNI 192.168.203.52@tcp [ 2853.151143] Key type .llcrypt unregistered [ 2853.153610] Key type ._llcrypt unregistered