[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 394396317 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003356] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.007761] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009023] pid_max: default: 32768 minimum: 301 [ 0.010194] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.013083] SELinux: Initializing. [ 0.015121] *** VALIDATE selinux *** [ 0.023432] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028609] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029205] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032040] *** VALIDATE tmpfs *** [ 0.033340] *** VALIDATE proc *** [ 0.035012] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.038029] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040074] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042037] Spectre V2 : User space: Vulnerable [ 0.043007] Speculative Store Bypass: Vulnerable [ 0.046709] debug: unmapping init [mem 0xffffffffb2459000-0xffffffffb2460fff] [ 0.048182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049782] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050023] ... version: 2 [ 0.051011] ... bit width: 48 [ 0.052009] ... generic registers: 4 [ 0.053010] ... value mask: 0000ffffffffffff [ 0.054012] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.056034] ... event mask: 000000070000000f [ 0.058184] rcu: Hierarchical SRCU implementation. [ 0.060446] smp: Bringing up secondary CPUs ... [ 0.061569] x86: Booting SMP configuration: [ 0.062012] .... node #0, CPUs: #1 #2 #3 [ 0.065612] smp: Brought up 1 node, 4 CPUs [ 0.066769] smpboot: Max logical packages: 1 [ 0.067010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.258563] node 0 deferred pages initialised in 188ms [ 0.264197] devtmpfs: initialized [ 0.265473] x86/mm: Memory block size: 128MB [ 0.269394] gcov: version magic: 0x41383552 [ 0.274590] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.278106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.279271] pinctrl core: initialized pinctrl subsystem [ 0.280099] [ 0.280648] ************************************************************* [ 0.283030] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285011] ** ** [ 0.288014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.290010] ** ** [ 0.293021] ** This means that this kernel is built to expose internal ** [ 0.295010] ** IOMMU data structures, which may compromise security on ** [ 0.297014] ** your system. ** [ 0.299012] ** ** [ 0.301010] ** If you see this message and you are not debugging the ** [ 0.303013] ** kernel, report this immediately to your vendor! ** [ 0.306012] ** ** [ 0.307006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.308008] ************************************************************* [ 0.310695] NET: Registered protocol family 16 [ 0.311510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322100] cpuidle: using governor menu [ 0.323499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325502] PCI: Using configuration type 1 for base access [ 0.328197] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.338136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.340039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342125] cryptd: max_cpu_qlen set to 1000 [ 0.344405] ACPI: Added _OSI(Module Device) [ 0.347070] ACPI: Added _OSI(Processor Device) [ 0.348011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.350015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.354904] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360555] ACPI: Interpreter enabled [ 0.361061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362010] ACPI: Using IOAPIC for interrupt routing [ 0.363190] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.366387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.375000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.375035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.377014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.380072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383978] acpiphp: Slot [2] registered [ 0.385147] acpiphp: Slot [3] registered [ 0.387058] acpiphp: Slot [4] registered [ 0.388054] acpiphp: Slot [5] registered [ 0.389112] acpiphp: Slot [6] registered [ 0.390088] acpiphp: Slot [7] registered [ 0.391083] acpiphp: Slot [8] registered [ 0.393121] acpiphp: Slot [9] registered [ 0.395103] acpiphp: Slot [10] registered [ 0.396082] acpiphp: Slot [11] registered [ 0.398109] acpiphp: Slot [12] registered [ 0.400133] acpiphp: Slot [13] registered [ 0.402208] acpiphp: Slot [14] registered [ 0.404073] acpiphp: Slot [15] registered [ 0.406066] acpiphp: Slot [16] registered [ 0.407124] acpiphp: Slot [17] registered [ 0.408090] acpiphp: Slot [18] registered [ 0.410076] acpiphp: Slot [19] registered [ 0.411062] acpiphp: Slot [20] registered [ 0.412065] acpiphp: Slot [21] registered [ 0.414072] acpiphp: Slot [22] registered [ 0.415087] acpiphp: Slot [23] registered [ 0.416062] acpiphp: Slot [24] registered [ 0.417091] acpiphp: Slot [25] registered [ 0.419042] acpiphp: Slot [26] registered [ 0.420051] acpiphp: Slot [27] registered [ 0.421000] acpiphp: Slot [28] registered [ 0.421000] acpiphp: Slot [29] registered [ 0.423090] acpiphp: Slot [30] registered [ 0.424073] acpiphp: Slot [31] registered [ 0.426058] PCI host bridge to bus 0000:00 [ 0.427012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.431020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.433015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.435018] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.438025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.448235] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.453544] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.457443] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.460015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.462012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.464300] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.467911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471612] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.474011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.481000] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.483893] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.486899] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497949] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517065] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.538170] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.552013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569038] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.581896] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.583335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.586395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.588333] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.590257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.595730] iommu: Default domain type: Passthrough [ 0.599687] SCSI subsystem initialized [ 0.600178] ACPI: bus type USB registered [ 0.603172] usbcore: registered new interface driver usbfs [ 0.605157] usbcore: registered new interface driver hub [ 0.608572] usbcore: registered new device driver usb [ 0.611242] pps_core: LinuxPPS API ver. 1 registered [ 0.612012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.616083] PTP clock support registered [ 0.619148] EDAC MC: Ver: 3.0.0 [ 0.621026] PCI: Using ACPI for IRQ routing [ 0.622460] NetLabel: Initializing [ 0.623000] NetLabel: domain hash size = 128 [ 0.625024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626149] NetLabel: unlabeled traffic allowed by default [ 0.630208] vgaarb: loaded [ 0.631350] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.633014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.638005] clocksource: Switched to clocksource kvm-clock [ 0.761695] VFS: Disk quotas dquot_6.6.0 [ 0.763098] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.765194] *** VALIDATE ramfs *** [ 0.766352] *** VALIDATE hugetlbfs *** [ 0.767894] pnp: PnP ACPI init [ 0.770532] pnp: PnP ACPI: found 6 devices [ 0.788139] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.791810] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.794101] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.796424] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.799060] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.801784] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.804781] NET: Registered protocol family 2 [ 0.807297] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.812803] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.818136] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.824742] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.828886] TCP: Hash tables configured (established 65536 bind 65536) [ 0.833422] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.836692] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839326] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.843253] NET: Registered protocol family 1 [ 0.848206] RPC: Registered named UNIX socket transport module. [ 0.850095] RPC: Registered udp transport module. [ 0.853779] RPC: Registered tcp transport module. [ 0.855213] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.858029] NET: Registered protocol family 44 [ 0.859504] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.861513] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.863345] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.865575] PCI: CLS 0 bytes, default 64 [ 0.867276] Unpacking initramfs... [ 2.848524] debug: unmapping init [mem 0xffff98e93cc64000-0xffff98e93ffcffff] [ 2.852250] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.854200] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.856989] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.733926] Initialise system trusted keyrings [ 3.735095] Key type blacklist registered [ 3.737354] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.744957] zbud: loaded [ 3.748382] *** VALIDATE nfs *** [ 3.749237] *** VALIDATE nfs4 *** [ 3.750275] pstore: using deflate compression [ 3.753872] Platform Keyring initialized [ 3.899802] NET: Registered protocol family 38 [ 3.901353] Key type asymmetric registered [ 3.902436] Asymmetric key parser 'x509' registered [ 3.903708] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.908923] io scheduler mq-deadline registered [ 3.910013] io scheduler kyber registered [ 3.911072] io scheduler bfq registered [ 3.912421] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.916651] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.918804] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.920908] ACPI: Power Button [PWRF] [ 4.022531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.128160] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.235864] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.269131] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.300606] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.307049] Non-volatile memory driver v1.3 [ 4.308389] Linux agpgart interface v0.103 [ 4.342680] virtio_blk virtio1: [vda] 133056 512-byte logical blocks (68.1 MB/65.0 MiB) [ 4.347254] vda: detected capacity change from 0 to 68124672 [ 4.365148] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.367903] vdb: detected capacity change from 0 to 1073741824 [ 4.384513] libphy: Fixed MDIO Bus: probed [ 4.398237] usbcore: registered new interface driver usbserial_generic [ 4.400421] usbserial: USB Serial support registered for generic [ 4.402633] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.407506] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.409220] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.411730] mousedev: PS/2 mouse device common for all mice [ 4.414757] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.419185] rtc_cmos 00:05: RTC can wake from S4 [ 4.425468] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.426127] rtc_cmos 00:05: registered as rtc0 [ 4.434865] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.437214] intel_pstate: CPU model not supported [ 4.440529] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.446514] hid: raw HID events driver (C) Jiri Kosina [ 4.448476] usbcore: registered new interface driver usbhid [ 4.450341] usbhid: USB HID core driver [ 4.451827] drop_monitor: Initializing network drop monitor service [ 4.457296] Initializing XFRM netlink socket [ 4.459135] NET: Registered protocol family 10 [ 4.466241] Segment Routing with IPv6 [ 4.467457] NET: Registered protocol family 17 [ 4.469809] mpls_gso: MPLS GSO support [ 4.474557] RAS: Correctable Errors collector initialized. [ 4.476347] AVX version of gcm_enc/dec engaged. [ 4.477796] AES CTR mode by8 optimization enabled [ 4.587676] sched_clock: Marking stable (4587651139, 0)->(5393365672, -805714533) [ 4.590892] registered taskstats version 1 [ 4.593660] Loading compiled-in X.509 certificates [ 4.595844] zswap: loaded using pool lzo/zbud [ 4.642472] Key type big_key registered [ 4.659026] Key type encrypted registered [ 4.660410] ima: No TPM chip found, activating TPM-bypass! [ 4.661915] ima: Allocated hash algorithm: sha1 [ 4.663164] ima: No architecture policies found [ 4.664431] evm: Initialising EVM extended attributes: [ 4.665704] evm: security.selinux [ 4.666526] evm: security.ima [ 4.667232] evm: security.capability [ 4.668045] evm: HMAC attrs: 0x1 [ 4.671468] rtc_cmos 00:05: setting system clock to 2025-09-02 10:33:48 UTC (1756809228) [ 4.677339] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 4.679707] debug: unmapping init [mem 0xffffffffb2182000-0xffffffffb2458fff] [ 4.687063] Write protecting the kernel read-only data: 28672k [ 4.690634] debug: unmapping init [mem 0xffffffffb0803000-0xffffffffb09fffff] [ 4.694286] debug: unmapping init [mem 0xffffffffb1114000-0xffffffffb11fffff] [ 4.730767] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.738565] systemd[1]: Detected virtualization kvm. [ 4.740370] systemd[1]: Detected architecture x86-64. [ 4.741994] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.771204] systemd[1]: No hostname configured. [ 4.773922] systemd[1]: Set hostname to . [ 4.777562] random: systemd: uninitialized urandom read (16 bytes read) [ 4.780962] systemd[1]: Initializing machine ID from random generator. [ 4.837594] random: ln: uninitialized urandom read (6 bytes read) [ 5.003810] random: systemd: uninitialized urandom read (16 bytes read) [ 5.008402] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.020535] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 5.038583] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.036477] device-mapper: uevent: version 1.0.3 [ 6.039166] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 7.186306] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.273696] virtio_net virtio0 ens2: renamed from eth0 [ 7.535892] scsi host0: ata_piix [ 7.603145] scsi host1: ata_piix [ 7.604453] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.624881] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.213955] random: crng init done [ 12.217832] random: 7 urandom warning(s) missed due to ratelimiting [ 15.508586] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.017415] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.032334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.483800] SELinux: Disabled at runtime. [ 20.551234] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.562767] systemd[1]: Detected virtualization kvm. [ 20.564944] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.851504] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.858431] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.884765] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.895889] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.905292] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.929102] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.946036] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward [ 22.293281] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.405864] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.616702] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.004123] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.111174] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.658311] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.828914] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.533255] NFS: Registering the id_resolver key type [ 30.534916] Key type id_resolver registered [ 30.543316] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg211-client login: [ 81.405922] libcfs: loading out-of-tree module taints kernel. [ 81.479576] Key type ._llcrypt registered [ 81.481310] Key type .llcrypt registered [ 81.773928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 81.781196] alg: No test for adler32 (adler32-zlib) [ 82.886395] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 83.275081] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 84.912141] Key type lgssc registered [ 85.487953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.402074] Lustre: Mounted lustre-client [ 148.207519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 160.512552] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing check_logdir /tmp/testlogs/ [ 162.762144] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing yml_node [ 164.980773] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 166.176586] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 167.421180] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 168.111807] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 2 06:36:31 EDT 2025 [ 170.976487] Lustre: lustre-OST0000-osc-ffff98e988f34000: disconnect after 24s idle [ 174.608538] LustreError: 7902:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff98e988f34000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 174.613277] LustreError: 7902:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 174.624682] Lustre: Unmounted lustre-client [ 189.821325] Key type lgssc unregistered [ 189.988527] LNet: 8348:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 189.991378] LNetError: 8348:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 190.001364] LNet: Removed LNI 192.168.202.11@tcp [ 190.418133] Key type .llcrypt unregistered [ 190.419906] Key type ._llcrypt unregistered [ 205.408260] Key type ._llcrypt registered [ 205.410151] Key type .llcrypt registered [ 205.505037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 205.513919] alg: No test for adler32 (adler32-zlib) [ 206.367860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 206.402556] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 206.419637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 208.596455] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 208.761149] LNet: 9220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.765746] LNetError: 9220:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 208.775826] LNet: Removed LNI 192.168.202.11@tcp [ 209.156599] Key type .llcrypt unregistered [ 209.158051] Key type ._llcrypt unregistered [ 212.409484] Key type ._llcrypt registered [ 212.411579] Key type .llcrypt registered [ 213.132761] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:37:16 (1756809436) [ 213.386479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 213.395874] alg: No test for adler32 (adler32-zlib) [ 214.236597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.288053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.299255] LNet: 9625:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 214.311781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.344209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 214.353824] LNet: 9632:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 218.490876] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:37:21 (1756809441) [ 218.618170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.632722] LNet: 10209:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 218.648497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.705571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 222.544457] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:37:25 (1756809445) [ 222.630414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.646434] LNet: 10793:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 222.670995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.720978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 226.381282] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:37:29 (1756809449) [ 226.426388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.439731] LNet: 11372:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 226.452567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.520835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 229.783941] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:37:32 (1756809452) [ 229.832274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.854534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.903233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 230.129270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 233.732444] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:37:36 (1756809456) [ 233.777562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.793488] LNet: 12545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.795694] LNet: 12545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 233.810830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.870360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 234.110718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 234.337658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 238.310753] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:37:41 (1756809461) [ 238.466641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.503523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.562860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 242.701193] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:37:45 (1756809465) [ 242.748813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.761831] LNet: 13722:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 242.764764] LNet: 13722:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 242.777977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.827289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 242.873594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 243.186661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 243.210451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 243.456537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 243.480956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 243.756029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 243.779523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 244.035865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 244.076930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 244.345569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 244.370419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 244.590809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 244.616777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 244.892187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 244.909283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 245.142850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 245.165474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 245.457575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 245.489825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 245.731193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 245.775545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 246.020906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 246.051282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 246.330380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 246.364727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 251.154468] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:37:53 (1756809473) [ 251.296218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.327585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.384683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 251.414144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 255.807818] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:37:58 (1756809478) [ 255.950563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.987837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.068470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 256.093507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 260.437277] hrtimer: interrupt took 13267896 ns [ 260.526525] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:38:03 (1756809483) [ 260.707127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.719213] LNet: 15583:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.722506] LNet: 15583:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 260.737161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.786406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 260.807315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 265.671744] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:38:08 (1756809488) [ 265.795452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.828161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.878430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 265.906304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 270.464408] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:38:13 (1756809493) [ 270.526491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.550594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.609724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 270.649807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 274.968820] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:38:17 (1756809497) [ 275.018539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.052929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.106486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 275.127956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 279.513947] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:38:22 (1756809502) [ 279.566239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.588116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.658501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 279.952066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 284.222256] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:38:27 (1756809507) [ 284.361205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.396861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.559884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 284.714267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 288.623041] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:38:31 (1756809511) [ 288.716871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.755162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.854500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.006944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 292.760469] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:38:35 (1756809515) [ 292.799596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.811226] LNet: 19747:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.815507] LNet: 19747:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 292.834970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.122246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 293.284289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 297.226797] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:38:40 (1756809520) [ 297.373244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.415422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.471068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.539659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 301.706833] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:38:44 (1756809524) [ 301.751298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.783577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.841224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.877753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 305.874737] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:38:48 (1756809528) [ 305.972262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.007556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.064910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 309.527669] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:38:52 (1756809532) [ 309.673707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.704663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.751269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 313.194984] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:38:56 (1756809536) [ 313.298859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.369216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 313.564679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 316.731559] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:38:59 (1756809539) [ 316.786580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.810722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.847539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 316.865726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 319.769411] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:39:03 (1756809543) [ 319.836072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.857773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.892401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 319.908584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 322.684974] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:39:05 (1756809545) [ 322.719909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.741068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.780039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 322.799079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 325.825502] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:39:08 (1756809548) [ 325.932790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.956261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.007726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 326.036410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 329.327399] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:39:12 (1756809552) [ 329.458834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.480493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.511790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 329.546361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 329.557200] LNetError: 25718:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 329.580928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 329.606845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 332.886366] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:39:15 (1756809555) [ 332.961779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.993424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.032313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NFjtVDhbYp [ 333.078522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NFjtVDhbYp [ 333.112239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NFjtVDhbYp [ 336.424358] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:39:19 (1756809559) [ 336.470654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.491567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.529474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 336.553889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 336.581429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 339.798935] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:39:22 (1756809562) [ 339.846880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.867631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.911773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 341.101312] LNet: Message forwarding will be enabled in 2s [ 344.161261] LNet: Message forwarding enabled [ 344.326459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 344.336885] LNet: Message forwarding will stop in 2s to 6s [ 348.576904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 349.132782] LNet: Message forwarding disabled [ 349.806322] LNet: Message forwarding will be enabled in 2s [ 353.194236] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:39:36 (1756809576) [ 353.251577] LNet: Message forwarding enabled [ 353.277088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.927961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.966606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 357.092458] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:39:40 (1756809580) [ 357.169571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.185765] LNet: 28744:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.189319] LNet: 28744:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 357.203552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.230823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 357.250408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 357.268229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 357.283781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 357.302661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 357.320612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 357.337831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 357.356945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 357.385432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 357.406150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 357.422850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 357.441798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 357.465420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 357.482706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 357.500206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 357.521085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 357.540989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 357.561745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 357.584244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 357.595832] LNetError: 28788:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 360.497972] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:39:43 (1756809583) [ 360.566886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.589395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.623382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.360531] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:39:46 (1756809586) [ 364.094631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg211-server [ 364.819455] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:39:48 (1756809588) [ 364.878097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.899146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.928950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 364.940610] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 365.696963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[12-16]@tcp [ 365.709427] LNet: 30500:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 365.727892] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.13@tcp is being used as a gateway but routing feature is not turned on [ 365.739629] LNetError: 30293:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.13@tcp has gone from up to down [ 366.760151] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.14@tcp is being used as a gateway but routing feature is not turned on [ 366.769460] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 367.784050] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.13@tcp is being used as a gateway but routing feature is not turned on [ 367.791385] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 368.376507] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:39:51 (1756809591) [ 368.802934] LNetError: 30297:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.12@tcp has gone from up to down [ 368.812842] LNetError: 30297:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 369.110355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg211-server [ 369.830336] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.16@tcp is being used as a gateway but routing feature is not turned on [ 369.840084] LNetError: 30293:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 369.841843] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:39:53 (1756809593) [ 369.888232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.848168] LNetError: 30292:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.13@tcp [ 371.937486] LNetError: 31428:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.949382] LNet: Removed LNI 192.168.202.11@tcp [ 371.965613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.992499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 372.007893] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 372.299298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[12-17/2]@tcp [ 372.309494] LNet: 31548:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 372.313510] LNet: 31548:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 372.323130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[12-17/2]@tcp [ 372.326175] LNetError: 31438:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.14@tcp has gone from up to down [ 372.335883] LNetError: 31438:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 374.905055] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:39:58 (1756809598) [ 374.980848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.992309] LNetError: 32140:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.001808] LNet: Removed LNI 192.168.202.11@tcp [ 375.010930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.051109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 375.078147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.094809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.125050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.152453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.169795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.194499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 375.229403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.245337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.273722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.307719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.327659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.354641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 375.389233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.407498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.441269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.476487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.493422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.520511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 375.556423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.575810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.604983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.636714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.653904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.677419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 375.710508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.726791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.755653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.788777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.807949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.837275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 375.861614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.880036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.913770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 378.568901] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:40:01 (1756809601) [ 379.478704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg211-server [ 380.205202] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:40:03 (1756809603) [ 381.054898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg211-server [ 381.873835] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:40:05 (1756809605) [ 382.249498] Key type .llcrypt unregistered [ 382.251368] Key type ._llcrypt unregistered [ 384.846023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 384.889322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 384.892255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 384.894635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 387.123916] Key type ._llcrypt registered [ 387.125462] Key type .llcrypt registered [ 387.201159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 387.209189] alg: No test for adler32 (adler32-zlib) [ 388.025570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.052682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 388.068583] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 388.345558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 388.355973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 389.235788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 389.252291] LNet: Removed LNI 10.1.2.3@tcp [ 389.299682] LNet: 34250:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.303283] LNetError: 34250:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.312471] LNet: Removed LNI 192.168.202.11@tcp [ 389.600165] Key type .llcrypt unregistered [ 389.601713] Key type ._llcrypt unregistered [ 393.886617] Key type ._llcrypt registered [ 393.887827] Key type .llcrypt registered [ 394.438622] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:40:17 (1756809617) [ 394.692510] Key type .llcrypt unregistered [ 394.693988] Key type ._llcrypt unregistered [ 397.077592] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 397.140625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 397.146168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 397.149113] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 399.371993] Key type ._llcrypt registered [ 399.373694] Key type .llcrypt registered [ 399.462001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 399.468435] alg: No test for adler32 (adler32-zlib) [ 400.291793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.323026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 400.335764] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 400.675207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 400.687187] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 401.888137] LNet: 35733:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.891504] LNetError: 35733:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.894988] LNet: Removed LNI 192.168.202.11@tcp [ 401.904447] LNet: Removed LNI 10.1.2.3@tcp [ 402.186922] Key type .llcrypt unregistered [ 402.188356] Key type ._llcrypt unregistered [ 406.698927] Key type ._llcrypt registered [ 406.700398] Key type .llcrypt registered [ 407.315366] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:40:30 (1756809630) [ 407.521541] Key type .llcrypt unregistered [ 407.523285] Key type ._llcrypt unregistered [ 410.058736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 410.105629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 410.115709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 410.117921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 412.390901] Key type ._llcrypt registered [ 412.392527] Key type .llcrypt registered [ 412.483075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.490781] alg: No test for adler32 (adler32-zlib) [ 413.310927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.339464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 413.357842] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 414.261963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 414.273342] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.097206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 415.107606] LNet: Removed LNI 10.1.2.3@tcp [ 415.115545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 415.132441] LNet: Removed LNI 10.1.2.31@tcp [ 415.141990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 415.156033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.499483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 415.512708] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 415.958948] LNet: 37639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.963485] LNetError: 37639:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 415.968810] LNet: Removed LNI 10.1.2.3@tcp [ 415.977506] LNet: Removed LNI 10.1.2.31@tcp [ 416.276517] Key type .llcrypt unregistered [ 416.277879] Key type ._llcrypt unregistered [ 421.444911] Key type ._llcrypt registered [ 421.450679] Key type .llcrypt registered [ 422.072413] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:40:45 (1756809645) [ 422.280765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 422.289237] alg: No test for adler32 (adler32-zlib) [ 423.132779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.163170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 423.179569] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 423.574522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.590138] LNet: 38584:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.595128] LNetError: 38584:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.604765] LNet: Removed LNI 192.168.202.11@tcp [ 423.619520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.656321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.11@tcp [ 423.669441] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 424.047284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.11@tcp [ 424.077097] LNet: Removed LNI 192.168.202.11@tcp [ 424.135443] LNet: 38716:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.477452] Key type .llcrypt unregistered [ 424.479289] Key type ._llcrypt unregistered [ 429.653233] Key type ._llcrypt registered [ 429.654859] Key type .llcrypt registered [ 430.257166] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:40:53 (1756809653) [ 431.112514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg211-server [ 431.879611] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:40:55 (1756809655) [ 432.198397] Key type .llcrypt unregistered [ 432.200123] Key type ._llcrypt unregistered [ 434.890607] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 434.927423] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 434.931805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 434.934493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 437.231418] Key type ._llcrypt registered [ 437.233104] Key type .llcrypt registered [ 437.316957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 437.323936] alg: No test for adler32 (adler32-zlib) [ 438.167886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.200986] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 438.545648] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 439.666334] LNet: 40535:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.673345] LNetError: 40535:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.676465] LNet: Removed LNI 192.168.202.11@tcp1 [ 439.690718] LNet: Removed LNI 10.1.2.3@tcp2 [ 440.132486] Key type .llcrypt unregistered [ 440.134258] Key type ._llcrypt unregistered [ 445.827842] Key type ._llcrypt registered [ 445.829510] Key type .llcrypt registered [ 446.531530] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:41:09 (1756809669) [ 446.779757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 446.789079] alg: No test for adler32 (adler32-zlib) [ 447.624533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.652523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 447.676893] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 448.124340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.137365] LNet: 41492:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.141418] LNetError: 41492:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 448.150564] LNet: Removed LNI 192.168.202.11@tcp [ 448.162511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.221494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 448.233481] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 451.445856] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:41:14 (1756809674) [ 451.565678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.585380] LNet: 42087:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.589981] LNetError: 42087:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.601659] LNet: Removed LNI 192.168.202.11@tcp [ 451.615879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.645374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 451.659852] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 452.134694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.11 [ 452.147064] LNet: Removed LNI 192.168.202.11@tcp [ 455.202759] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:41:18 (1756809678) [ 455.290669] LNet: 42785:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.298712] LNetError: 42785:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.310397] LNet: Removed LNI 192.168.202.11@tcp [ 455.628060] Key type .llcrypt unregistered [ 455.629732] Key type ._llcrypt unregistered [ 458.408782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 458.519377] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 458.521678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 458.524291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 460.859951] Key type ._llcrypt registered [ 460.861577] Key type .llcrypt registered [ 460.931577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 460.938158] alg: No test for adler32 (adler32-zlib) [ 461.955682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 465.307530] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:41:28 (1756809688) [ 465.388937] LNet: 43685:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.400355] LNetError: 43685:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 465.414772] LNet: Removed LNI 10.1.2.3@tcp [ 465.759526] Key type .llcrypt unregistered [ 465.760976] Key type ._llcrypt unregistered [ 468.754341] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 468.865855] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 468.868444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 468.871517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 471.239377] Key type ._llcrypt registered [ 471.244480] Key type .llcrypt registered [ 471.336216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 471.344258] alg: No test for adler32 (adler32-zlib) [ 472.196076] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 475.416166] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:41:38 (1756809698) [ 475.512033] LNet: 44574:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.519709] LNetError: 44574:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 475.531781] LNet: Removed LNI 10.1.2.3@tcp [ 475.842453] Key type .llcrypt unregistered [ 475.843888] Key type ._llcrypt unregistered [ 478.915787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 479.018789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 479.021644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 479.024902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 481.358312] Key type ._llcrypt registered [ 481.359945] Key type .llcrypt registered [ 481.439905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 481.449486] alg: No test for adler32 (adler32-zlib) [ 482.300320] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 485.845841] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:41:48 (1756809708) [ 485.981603] LNet: 45463:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.985721] LNetError: 45463:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 485.998784] LNet: Removed LNI 10.1.2.3@tcp [ 486.310510] Key type .llcrypt unregistered [ 486.312190] Key type ._llcrypt unregistered [ 489.227663] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 489.339422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 489.341584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 489.344585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 491.723363] Key type ._llcrypt registered [ 491.725410] Key type .llcrypt registered [ 491.820463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 491.830204] alg: No test for adler32 (adler32-zlib) [ 492.661150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.708107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 496.992140] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:42:00 (1756809720) [ 497.276878] LNet: 46684:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.280776] LNetError: 46684:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 497.295038] LNet: Removed LNI 10.1.2.3@tcp [ 497.641500] Key type .llcrypt unregistered [ 497.643258] Key type ._llcrypt unregistered [ 500.162333] Key type ._llcrypt registered [ 500.163605] Key type .llcrypt registered [ 500.266404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 500.275928] alg: No test for adler32 (adler32-zlib) [ 501.107435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 501.130776] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 504.928905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 505.230617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_interrupt -r 1 [ 505.249225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_interrupt -r 1 [ 505.265518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 505.275743] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 505.410160] LNet: 47109:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 505.445025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 505.461848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 505.483825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 505.702806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_dropped -r 1 [ 505.719923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_dropped -r 1 [ 505.737449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 505.906297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 505.924079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 505.940333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 506.173277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_aborted -r 1 [ 506.190614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_aborted -r 1 [ 506.211603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 506.225191] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 506.227867] LNet: Skipped 1 previous similar message [ 506.357909] LNet: 47303:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 506.361735] LNet: 47303:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 506.396427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 506.415658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 506.434259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 506.681672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_no_route -r 1 [ 506.699677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_no_route -r 1 [ 506.717527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 506.889310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 506.907257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 506.925184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 507.140491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_timeout -r 1 [ 507.162834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_timeout -r 1 [ 507.185481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 507.361754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 507.379451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 507.403551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 507.635751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_error -r 1 [ 507.654183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_error -r 1 [ 507.671569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 507.683404] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 507.686713] LNet: Skipped 2 previous similar messages [ 507.815094] LNet: 47594:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 507.818553] LNet: 47594:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 507.854837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 507.875949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 507.893384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 508.004774] LNet: 47630:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.008365] LNetError: 47630:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.020981] LNet: Removed LNI 192.168.202.11@tcp [ 508.198500] Key type .llcrypt unregistered [ 508.199922] Key type ._llcrypt unregistered [ 515.083675] Key type ._llcrypt registered [ 515.085275] Key type .llcrypt registered [ 515.683842] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:42:18 (1756809738) [ 516.101658] Key type .llcrypt unregistered [ 516.104079] Key type ._llcrypt unregistered [ 518.620961] Key type ._llcrypt registered [ 518.623114] Key type .llcrypt registered [ 518.708977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 518.716353] alg: No test for adler32 (adler32-zlib) [ 519.528844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 519.549697] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 522.953950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 526.963927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.980725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 526.992977] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 527.550446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_interrupt -r 1 [ 527.569905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_interrupt -r 1 [ 527.590857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_interrupt -r 1 [ 527.611188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_interrupt -r 1 [ 527.628872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_interrupt -r 1 [ 527.646610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_interrupt -r 1 [ 527.663325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_interrupt -r 1 [ 527.680759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_interrupt -r 1 [ 527.698903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 527.709037] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 527.712974] LNet: 1 local NIs in recovery (showing 1): 192.168.202.11@tcp [ 527.844738] LNet: 49076:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 527.877914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 527.894483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 527.912682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 528.114692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_dropped -r 1 [ 528.132785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_dropped -r 1 [ 528.153548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_dropped -r 1 [ 528.175568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_dropped -r 1 [ 528.197113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_dropped -r 1 [ 528.216725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_dropped -r 1 [ 528.236534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_dropped -r 1 [ 528.256492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_dropped -r 1 [ 528.274708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 528.284066] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 528.287228] LNet: Skipped 2 previous similar messages [ 528.410131] LNet: 49185:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 528.439976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 528.471590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 528.491365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 528.729637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_aborted -r 1 [ 528.747328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_aborted -r 1 [ 528.768330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_aborted -r 1 [ 528.788320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_aborted -r 1 [ 528.810673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_aborted -r 1 [ 528.828716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_aborted -r 1 [ 528.846846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_aborted -r 1 [ 528.867408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_aborted -r 1 [ 528.886678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 529.048969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 529.065609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 529.083063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 529.298036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_no_route -r 1 [ 529.314954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_no_route -r 1 [ 529.332985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_no_route -r 1 [ 529.351514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_no_route -r 1 [ 529.370924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_no_route -r 1 [ 529.392566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_no_route -r 1 [ 529.413583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_no_route -r 1 [ 529.431771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_no_route -r 1 [ 529.450746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 529.460989] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 529.464678] LNet: Skipped 6 previous similar messages [ 529.597375] LNet: 49404:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 529.600460] LNet: 49404:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 529.634562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 529.650311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 529.665950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 529.878704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_timeout -r 1 [ 529.896475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_timeout -r 1 [ 529.920085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_timeout -r 1 [ 529.943218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_timeout -r 1 [ 529.961962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_timeout -r 1 [ 529.980298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_timeout -r 1 [ 530.001814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_timeout -r 1 [ 530.024681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_timeout -r 1 [ 530.043808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 530.217390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 530.233903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 530.251625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 530.479607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_error -r 1 [ 530.496686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_error -r 1 [ 530.513810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_error -r 1 [ 530.532349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_error -r 1 [ 530.550663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_error -r 1 [ 530.569966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_error -r 1 [ 530.588454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_error -r 1 [ 530.604328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_error -r 1 [ 530.620328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 530.782819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 530.799785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 530.818701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 533.245723] LNet: 49692:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.249981] LNetError: 49692:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.254435] LNet: Removed LNI 192.168.202.11@tcp [ 534.311692] LNet: Removed LNI 192.168.202.11@tcp1 [ 534.521785] Key type .llcrypt unregistered [ 534.523560] Key type ._llcrypt unregistered [ 542.096810] Key type ._llcrypt registered [ 542.098837] Key type .llcrypt registered [ 542.832157] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:42:45 (1756809765) [ 543.249379] Key type .llcrypt unregistered [ 543.251097] Key type ._llcrypt unregistered [ 545.631944] Key type ._llcrypt registered [ 545.633054] Key type .llcrypt registered [ 545.716358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 545.724407] alg: No test for adler32 (adler32-zlib) [ 546.545953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 546.560157] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 550.029685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 550.331991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_dropped -r 1 [ 550.350647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_dropped -r 1 [ 550.371473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 550.383237] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 550.528315] LNet: 50924:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 550.564774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 550.589485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 550.609170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 550.857373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_error -r 1 [ 550.879807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_error -r 1 [ 550.898365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 550.909886] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 551.056255] LNet: 51021:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 551.095409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 551.111892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 551.132421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 551.339983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_timeout -r 1 [ 551.357489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_timeout -r 1 [ 551.374066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 551.560268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 551.574377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 551.590538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 551.705341] LNet: 51154:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.708476] LNetError: 51154:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.743046] LNet: Removed LNI 192.168.202.11@tcp [ 552.967217] Key type .llcrypt unregistered [ 552.970791] Key type ._llcrypt unregistered [ 560.125756] Key type ._llcrypt registered [ 560.127293] Key type .llcrypt registered [ 560.743904] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:43:03 (1756809783) [ 561.114291] Key type .llcrypt unregistered [ 561.115698] Key type ._llcrypt unregistered [ 563.468170] Key type ._llcrypt registered [ 563.469861] Key type .llcrypt registered [ 563.550143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.557404] alg: No test for adler32 (adler32-zlib) [ 564.364934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 564.376910] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 568.057291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 571.673995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.691532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 571.706226] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 572.325333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_dropped -r 1 [ 572.343776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e remote_dropped -r 1 [ 572.366405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_dropped -r 1 [ 572.390339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e remote_dropped -r 1 [ 572.412437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e remote_dropped -r 1 [ 572.435022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e remote_dropped -r 1 [ 572.456446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e remote_dropped -r 1 [ 572.479440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e remote_dropped -r 1 [ 572.500408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 572.512221] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 572.517661] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.111@tcp [ 572.662293] LNet: 52599:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 572.698529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 572.713606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 572.728665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 572.745182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 572.988510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_error -r 1 [ 573.006969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e remote_error -r 1 [ 573.026377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_error -r 1 [ 573.043862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e remote_error -r 1 [ 573.060211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e remote_error -r 1 [ 573.081438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e remote_error -r 1 [ 573.103160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e remote_error -r 1 [ 573.125327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e remote_error -r 1 [ 573.144768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 573.155390] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 573.158963] LNet: Skipped 2 previous similar messages [ 573.282189] LNet: 52710:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 573.315420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.331107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 573.347708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 573.363971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.588117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_timeout -r 1 [ 573.608061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e remote_timeout -r 1 [ 573.629074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_timeout -r 1 [ 573.648134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e remote_timeout -r 1 [ 573.668208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e remote_timeout -r 1 [ 573.686162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e remote_timeout -r 1 [ 573.705420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e remote_timeout -r 1 [ 573.725743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e remote_timeout -r 1 [ 573.747445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.111@tcp [ 573.935387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.953062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 573.970844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 573.985562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 576.447349] LNet: 52895:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.451742] LNetError: 52895:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.456544] LNet: Removed LNI 192.168.202.11@tcp [ 576.462693] LNet: Removed LNI 192.168.202.11@tcp1 [ 576.652267] Key type .llcrypt unregistered [ 576.654187] Key type ._llcrypt unregistered [ 583.752336] Key type ._llcrypt registered [ 583.753963] Key type .llcrypt registered [ 584.394465] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:43:27 (1756809807) [ 584.627416] Key type .llcrypt unregistered [ 584.629745] Key type ._llcrypt unregistered [ 586.881851] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 586.915354] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 586.919567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 586.922239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 589.175778] Key type ._llcrypt registered [ 589.177921] Key type .llcrypt registered [ 589.246395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 589.253837] alg: No test for adler32 (adler32-zlib) [ 590.083768] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 590.100212] LNet: 53978:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.103134] LNetError: 53978:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.112535] LNet: Removed LNI 192.168.202.11@tcp [ 590.416413] Key type .llcrypt unregistered [ 590.418042] Key type ._llcrypt unregistered [ 592.886484] Key type ._llcrypt registered [ 592.887614] Key type .llcrypt registered [ 592.978206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.985482] alg: No test for adler32 (adler32-zlib) [ 593.825331] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 593.827576] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 593.850646] LNet: 54246:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.854830] LNetError: 54246:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.859629] LNet: Removed LNI 192.168.202.11@tcp [ 593.868502] LNet: Removed LNI 10.1.2.3@tcp1 [ 594.201719] Key type .llcrypt unregistered [ 594.206759] Key type ._llcrypt unregistered [ 596.791780] Key type ._llcrypt registered [ 596.793479] Key type .llcrypt registered [ 596.861988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 596.868421] alg: No test for adler32 (adler32-zlib) [ 597.695425] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 597.697461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 597.719820] LNet: 54515:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.724751] LNetError: 54515:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.728816] LNet: Removed LNI 192.168.202.11@tcp [ 597.739957] LNet: Removed LNI 10.1.2.3@tcp [ 598.033426] Key type .llcrypt unregistered [ 598.034824] Key type ._llcrypt unregistered [ 600.491637] Key type ._llcrypt registered [ 600.493191] Key type .llcrypt registered [ 600.581521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 600.590536] alg: No test for adler32 (adler32-zlib) [ 601.423133] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 601.425103] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 601.444525] LNet: 54785:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.448131] LNetError: 54785:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.451565] LNet: Removed LNI 192.168.202.11@tcp [ 601.460057] LNet: Removed LNI 10.1.2.3@tcp [ 601.767505] Key type .llcrypt unregistered [ 601.769302] Key type ._llcrypt unregistered [ 604.222621] Key type ._llcrypt registered [ 604.224138] Key type .llcrypt registered [ 604.301127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.315475] alg: No test for adler32 (adler32-zlib) [ 605.142727] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 605.145302] LNetError: 55044:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 605.148817] LNetError: Error -100 starting up LNI tcp [ 605.159908] LNet: Removed LNI 192.168.202.11@tcp [ 605.483349] Key type .llcrypt unregistered [ 605.484625] Key type ._llcrypt unregistered [ 609.745097] Key type ._llcrypt registered [ 609.746483] Key type .llcrypt registered [ 610.292121] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:43:53 (1756809833) [ 610.613648] Key type .llcrypt unregistered [ 610.614990] Key type ._llcrypt unregistered [ 612.904241] Key type ._llcrypt registered [ 612.906318] Key type .llcrypt registered [ 613.032757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 613.039158] alg: No test for adler32 (adler32-zlib) [ 613.896847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 613.920920] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 618.014830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 618.338339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e network_timeout -r 1 [ 618.357272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e network_timeout -r 1 [ 618.376801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 618.389233] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 618.533555] LNet: 56282:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 618.572493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.590468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.709643] LNet: 56316:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.716325] LNetError: 56316:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.730205] LNet: Removed LNI 192.168.202.11@tcp [ 618.924729] Key type .llcrypt unregistered [ 618.926314] Key type ._llcrypt unregistered [ 626.882356] Key type ._llcrypt registered [ 626.884014] Key type .llcrypt registered [ 626.967982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 626.975319] alg: No test for adler32 (adler32-zlib) [ 627.815252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 627.833490] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 631.629120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 635.382236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.399108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 635.416032] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 636.046641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e network_timeout -r 1 [ 636.067311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e network_timeout -r 1 [ 636.086038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e network_timeout -r 1 [ 636.108550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e network_timeout -r 1 [ 636.130359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e network_timeout -r 1 [ 636.157322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e network_timeout -r 1 [ 636.180459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e network_timeout -r 1 [ 636.202860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e network_timeout -r 1 [ 636.225199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 636.240351] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 636.392967] LNet: 57175:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 636.433677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 636.449678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 638.907912] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 639.034427] LNet: 57243:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.037818] LNetError: 57243:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.041524] LNet: Removed LNI 192.168.202.11@tcp [ 639.050637] LNet: Removed LNI 192.168.202.11@tcp1 [ 639.217421] Key type .llcrypt unregistered [ 639.219533] Key type ._llcrypt unregistered [ 645.342701] Key type ._llcrypt registered [ 645.344109] Key type .llcrypt registered [ 645.891943] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:44:29 (1756809869) [ 646.036924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.043398] alg: No test for adler32 (adler32-zlib) [ 646.855349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.881424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.896277] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 647.194578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 647.206257] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 647.554783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 647.585209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 647.607268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 647.624764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 647.641767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 647.657906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 647.672755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.11@tcp [ 647.682359] LNetError: 58331:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.202.11@tcp to recovery [ 647.904346] LNet: 1 local NIs in recovery (showing 1): 192.168.202.11@tcp [ 647.904463] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 649.952325] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 650.864681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 653.024630] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 654.052033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 654.065828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 654.091028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.099962] LNet: 58438:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.103180] LNetError: 58438:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.106431] LNet: Removed LNI 192.168.202.11@tcp [ 654.113484] LNet: Removed LNI 192.168.202.11@tcp1 [ 654.123585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.150327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.162835] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 654.459750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 654.469571] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 654.805896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 654.824302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 654.850572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 654.873341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 654.891366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 654.909041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 654.925667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 654.942488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.11@tcp [ 654.955469] LNetError: 58704:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.202.11@tcp to recovery [ 655.136320] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 658.136274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 661.280341] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 661.283447] LNet: Skipped 1 previous similar message [ 665.478711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 665.495818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 665.512534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 668.279160] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:44:51 (1756809891) [ 668.313835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.323982] LNet: 59416:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.326959] LNetError: 59416:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.330436] LNetError: 59416:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 668.334827] LNet: Removed LNI 192.168.202.11@tcp [ 668.342592] LNet: Removed LNI 192.168.202.11@tcp1 [ 668.353741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.382548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.394995] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 668.701544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 668.712165] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 669.073462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 669.105040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 669.121830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 669.141629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 669.158243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 669.175792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 669.197851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.11@tcp --health 0 [ 669.248488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 669.408328] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.11@tcp [ 670.434570] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 670.437254] LNet: Skipped 1 previous similar message [ 684.795765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 684.854712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 684.869119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 684.884271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.11@tcp --health 500 [ 684.927263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 691.166849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 691.195954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.205771] LNet: 59913:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.209953] LNetError: 59913:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.213623] LNetError: 59913:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 691.217359] LNet: Removed LNI 192.168.202.11@tcp [ 691.230345] LNet: Removed LNI 192.168.202.11@tcp1 [ 691.239857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.264381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.276661] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 691.583948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 691.593920] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 691.940367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 691.957562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 691.985097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 691.999733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 692.016634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 692.033464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 692.050137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 692.066803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.11@tcp --health 0 [ 693.280351] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 693.283372] LNet: Skipped 3 previous similar messages [ 694.216719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 703.657845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 703.675541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 703.692925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 706.627537] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:45:29 (1756809929) [ 706.808459] LNet: 60965:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.811715] LNetError: 60965:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.815301] LNetError: 60965:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 706.818583] LNet: Removed LNI 192.168.202.11@tcp [ 706.824416] LNet: Removed LNI 192.168.202.11@tcp1 [ 707.107840] Key type .llcrypt unregistered [ 707.109424] Key type ._llcrypt unregistered [ 709.476877] Key type ._llcrypt registered [ 709.478630] Key type .llcrypt registered [ 709.564790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.572155] alg: No test for adler32 (adler32-zlib) [ 710.380393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 710.393168] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 714.378535] Lustre: DEBUG MARKER: Initial discovery [ 714.460507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.111@tcp [ 715.465334] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 715.533434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 715.552313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.111@tcp [ 715.564267] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 715.574118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 715.590657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.111@tcp -r 1 -m GET -l 3 [ 715.610797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.111@tcp -r 1 -m GET -e local_error [ 715.630195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.111@tcp -r 1 -m PUT -l 6 [ 716.215543] Lustre: DEBUG MARKER: Force oleg211-client.virtnet to discover 192.168.202.111@tcp (in background) [ 716.272868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.111@tcp [ 716.863931] Lustre: DEBUG MARKER: Force oleg211-server to discover 192.168.202.11@tcp [ 717.937391] Lustre: DEBUG MARKER: Wait for 61718 [ 719.329214] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 719.897488] Lustre: DEBUG MARKER: Finished wait on 61718 [ 719.927454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 719.946435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 720.054780] LNet: 62169:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.057809] LNetError: 62169:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.067594] LNet: Removed LNI 192.168.202.11@tcp [ 720.240668] Key type .llcrypt unregistered [ 720.242064] Key type ._llcrypt unregistered [ 726.388692] Key type ._llcrypt registered [ 726.390248] Key type .llcrypt registered [ 726.933476] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:45:50 (1756809950) [ 727.143864] Key type .llcrypt unregistered [ 727.145215] Key type ._llcrypt unregistered [ 729.424406] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 729.494225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 729.496587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 729.499443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.729603] Key type ._llcrypt registered [ 731.731087] Key type .llcrypt registered [ 731.814076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.821649] alg: No test for adler32 (adler32-zlib) [ 732.642278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.671222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.693126] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 733.034954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 733.047372] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 736.573481] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:45:59 (1756809959) [ 736.645182] LNet: 64236:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.648890] LNetError: 64236:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.652416] LNet: Removed LNI 192.168.202.11@tcp [ 736.660446] LNet: Removed LNI 10.1.2.3@tcp [ 736.927457] Key type .llcrypt unregistered [ 736.929090] Key type ._llcrypt unregistered [ 739.404395] Key type ._llcrypt registered [ 739.406691] Key type .llcrypt registered [ 739.497756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.504479] alg: No test for adler32 (adler32-zlib) [ 740.322268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.348686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.362804] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 740.648659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.659658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.072635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.11@tcp [ 742.112219] LNetError: Unexpected error -22 connecting to 192.168.202.11@tcp at host 192.168.202.11:988 [ 743.815034] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:46:07 (1756809967) [ 743.888188] LNet: 65355:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.891441] LNetError: 65355:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.894155] LNet: Removed LNI 192.168.202.11@tcp [ 743.902754] LNet: Removed LNI 10.1.2.3@tcp [ 744.167659] Key type .llcrypt unregistered [ 744.169342] Key type ._llcrypt unregistered [ 746.564437] Key type ._llcrypt registered [ 746.566365] Key type .llcrypt registered [ 746.661765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.669380] alg: No test for adler32 (adler32-zlib) [ 747.491647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.521791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 747.535966] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 747.874835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 747.889773] LNet: Added LNI 192.168.202.11@tcp2 [8/256/0/180] [ 748.295765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.11@tcp1 --nid 192.168.202.11@tcp2 [ 750.214949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp1 192.168.202.11@tcp2 [ 750.227303] LNetError: 66181:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.11@tcp1 to 192.168.202.11@tcp2 - Route table may be misconfigured [ 750.238909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp2 192.168.202.11@tcp1 [ 752.889719] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:46:16 (1756809976) [ 752.986932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.997321] LNet: 66753:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.001055] LNetError: 66753:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.004714] LNet: Removed LNI 192.168.202.11@tcp1 [ 754.022718] LNet: Removed LNI 192.168.202.11@tcp2 [ 754.033854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.062762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.077885] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 754.371570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 754.382026] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 754.721538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 754.738951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.11@tcp [ 754.755200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp1 192.168.202.11@tcp1 [ 754.775272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.11@tcp -d 192.168.202.11@tcp -e network_timeout [ 754.794641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -e network_timeout [ 754.812498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -e network_timeout [ 754.834336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -e network_timeout [ 754.852544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 754.862886] LNet: There was an unexpected network error while writing to 192.168.202.11: rc = -22 [ 754.872889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 754.915189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 754.954847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 755.106210] LNet: 1 local NIs in recovery (showing 1): 192.168.202.11@tcp [ 757.547758] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:46:20 (1756809980) [ 757.623785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.635316] LNet: 67606:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.638707] LNetError: 67606:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.641877] LNetError: 67606:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 757.645692] LNet: Removed LNI 192.168.202.11@tcp [ 758.696401] LNet: Removed LNI 192.168.202.11@tcp1 [ 758.715422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.765882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 758.890722] LNet: 67644:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.068352] Key type .llcrypt unregistered [ 759.070073] Key type ._llcrypt unregistered [ 763.491895] Key type ._llcrypt registered [ 763.493749] Key type .llcrypt registered [ 764.127664] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:46:27 (1756809987) [ 764.276425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.283912] alg: No test for adler32 (adler32-zlib) [ 765.113505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.735095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 766.403386] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:46:29 (1756809989) [ 766.455576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.467545] LNet: 68735:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.479731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.505576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.518938] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 766.838783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 766.850118] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 767.234995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 767.255997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp1 [ 767.275612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp1 [ 769.936892] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:46:33 (1756809993) [ 770.718619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg211-server [ 771.404766] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:46:34 (1756809994) [ 772.133812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg211-server [ 772.805803] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:46:36 (1756809996) [ 773.615151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg211-server [ 774.300298] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:46:37 (1756809997) [ 775.056375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg211-server [ 775.807765] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:46:39 (1756809999) [ 776.588609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg211-server [ 777.303790] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:46:40 (1756810000) [ 778.055900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg211-server [ 778.785622] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:46:42 (1756810002) [ 779.577417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg211-server [ 780.265410] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:46:43 (1756810003) [ 780.988906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg211-server [ 781.701567] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:46:44 (1756810004) [ 782.494466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg211-server [ 783.187242] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:46:46 (1756810006) [ 783.260747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.271606] LNet: 72604:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.274727] LNetError: 72604:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.278393] LNet: Removed LNI 192.168.202.11@tcp [ 784.294497] LNet: Removed LNI 192.168.202.11@tcp1 [ 784.306248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.333224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.346172] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 784.643560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 784.681372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 784.753027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.762613] LNet: 72748:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.765814] LNetError: 72748:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.768713] LNetError: 72748:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.777480] LNet: Removed LNI 192.168.202.11@tcp [ 784.787698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.814761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 784.829428] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 785.148418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 785.225778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.278496] LNet: Removed LNI 192.168.202.11@tcp [ 786.289425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.315773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.327078] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 786.617264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 786.656850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 786.734981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.748976] LNetError: 73034:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.752372] LNetError: 73034:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.765496] LNet: Removed LNI 192.168.202.11@tcp [ 786.775624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.804905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 786.817209] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 787.126050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 787.219684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.230990] LNet: 73176:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.233810] LNet: 73176:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 788.262311] LNet: Removed LNI 192.168.202.11@tcp [ 788.273654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.299616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.313371] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 788.604716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 788.640577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 788.726178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.798979] LNet: Removed LNI 192.168.202.11@tcp [ 789.809053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.835299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 789.848402] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 790.157590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 790.239993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.252672] LNetError: 73464:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.256274] LNetError: 73464:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 790.266324] LNet: Removed LNI 192.168.202.11@tcp [ 790.276471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.303811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 790.315440] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 790.611622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 790.649295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 790.732966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.783708] LNet: Removed LNI 192.168.202.11@tcp [ 791.793530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.823807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 791.835613] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 792.164217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 793.337829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.350500] LNet: 73759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.353969] LNet: 73759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 794.406592] LNet: Removed LNI 192.168.202.11@tcp [ 794.422571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.456808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.469398] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 794.773392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 794.806939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 794.917611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.927603] LNetError: 73903:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.931065] LNetError: 73903:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 794.937410] LNet: Removed LNI 192.168.202.11@tcp [ 794.946750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.973506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 794.986800] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 795.309670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 795.405252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.454740] LNet: Removed LNI 192.168.202.11@tcp [ 796.469479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.499522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.512597] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 796.808063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 796.842148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 796.942261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.991493] LNet: Removed LNI 192.168.202.11@tcp [ 798.004055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.031797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 798.046329] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 798.374830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 798.485498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.526502] LNet: Removed LNI 192.168.202.11@tcp [ 799.536071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.561884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.574356] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 799.869353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 799.896426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.918368] LNet: Removed LNI 192.168.202.11@tcp [ 799.927271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.951764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.965482] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 800.264910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 803.000214] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:47:06 (1756810026) [ 803.061294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.070662] LNet: 75160:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.073853] LNet: 75160:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 803.078410] LNetError: 75160:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.080934] LNetError: 75160:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 803.090018] LNet: Removed LNI 192.168.202.11@tcp [ 803.098432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.122164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 803.134443] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/180] [ 803.606420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.621432] LNet: Removed LNI 192.168.202.11@tcp231 [ 803.629835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.661221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 803.675767] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/180] [ 803.678364] LNet: 75305:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 803.865163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 803.887914] LNet: Removed LNI 192.168.202.11@tcp231 [ 803.895926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 803.910161] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/180] [ 804.406141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.421921] LNet: Removed LNI 192.168.202.11@tcp231 [ 804.433279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.468287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 804.481673] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/0] [ 804.484703] LNet: 75454:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 804.657630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 804.678583] LNet: Removed LNI 192.168.202.11@tcp231 [ 804.687762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 804.703310] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/0] [ 805.197618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.212572] LNet: Removed LNI 192.168.202.11@tcp231 [ 805.223747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.254875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 805.268416] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/60] [ 805.446723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 805.466904] LNet: Removed LNI 192.168.202.11@tcp231 [ 805.477274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 805.489854] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/60] [ 805.953257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.969694] LNet: Removed LNI 192.168.202.11@tcp231 [ 805.980309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.013759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 806.026353] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/180] [ 806.029464] LNet: 75750:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 806.034076] LNet: 75750:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 806.204178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 806.230563] LNet: Removed LNI 192.168.202.11@tcp231 [ 806.238765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 806.251240] LNet: Added LNI 192.168.202.11@tcp231 [8/256/0/180] [ 809.430356] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:47:12 (1756810032) [ 809.480915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.496059] LNet: Removed LNI 192.168.202.11@tcp231 [ 809.507738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.541465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 809.554978] LNet: Added LNI 192.168.202.11@tcp232 [8/256/0/180] [ 809.868624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 809.896394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 809.920426] LNet: Removed LNI 192.168.202.11@tcp232 [ 809.928558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 809.940612] LNet: Added LNI 192.168.202.11@tcp232 [8/256/0/180] [ 810.291658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 810.318397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 810.336695] LNet: Removed LNI 192.168.202.11@tcp232 [ 810.345242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 810.360106] LNet: Added LNI 192.168.202.11@tcp232 [8/256/0/180] [ 810.727801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 810.744971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 810.763460] LNet: Removed LNI 192.168.202.11@tcp232 [ 810.775153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 810.788567] LNet: Added LNI 192.168.202.11@tcp232 [8/256/0/180] [ 811.104304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 811.124296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 811.141649] LNet: Removed LNI 192.168.202.11@tcp232 [ 811.153040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 814.655491] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:47:17 (1756810037) [ 815.772702] Key type .llcrypt unregistered [ 815.775614] Key type ._llcrypt unregistered [ 821.532146] Key type ._llcrypt registered [ 821.534088] Key type .llcrypt registered [ 821.891938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.922687] alg: No test for adler32 (adler32-zlib) [ 823.085314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 823.146598] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 831.047842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 838.436530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.499116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 838.533793] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 840.389496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_interrupt -r 2 [ 840.437648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_interrupt -r 2 [ 840.490433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_interrupt -r 2 [ 840.559949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_interrupt -r 2 [ 840.611246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_interrupt -r 2 [ 840.662755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_interrupt -r 2 [ 840.705199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_interrupt -r 2 [ 840.737199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_interrupt -r 2 [ 840.768987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 840.790821] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 840.797950] LNet: 1 local NIs in recovery (showing 1): 192.168.202.11@tcp [ 841.175584] LNet: 78218:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 841.263706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 841.353437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 842.115344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_dropped -r 2 [ 842.171476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_dropped -r 2 [ 842.224481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_dropped -r 2 [ 842.271324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_dropped -r 2 [ 842.300883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_dropped -r 2 [ 842.360962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_dropped -r 2 [ 842.446889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_dropped -r 2 [ 842.549294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_dropped -r 2 [ 842.631677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 842.693701] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 843.167669] LNet: 78326:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 843.254245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 843.279899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 843.851531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_aborted -r 2 [ 843.893763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_aborted -r 2 [ 843.967743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_aborted -r 2 [ 844.025905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_aborted -r 2 [ 844.072613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_aborted -r 2 [ 844.117853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_aborted -r 2 [ 844.176702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_aborted -r 2 [ 844.238138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_aborted -r 2 [ 844.272355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 844.296934] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 844.627335] LNet: 78434:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 844.711855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.765536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 845.381178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_no_route -r 2 [ 845.408947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_no_route -r 2 [ 845.433609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_no_route -r 2 [ 845.488859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_no_route -r 2 [ 845.525690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_no_route -r 2 [ 845.567127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_no_route -r 2 [ 845.600799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_no_route -r 2 [ 845.663688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_no_route -r 2 [ 845.740626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 846.329360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 846.354313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 846.915590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e local_timeout -r 2 [ 846.951710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e local_timeout -r 2 [ 847.015769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e local_timeout -r 2 [ 847.090199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e local_timeout -r 2 [ 847.169252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e local_timeout -r 2 [ 847.227685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e local_timeout -r 2 [ 847.271147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e local_timeout -r 2 [ 847.352571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e local_timeout -r 2 [ 847.373545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 847.387441] LNet: There was an unexpected network error while writing to 192.168.202.111: rc = -22 [ 847.390435] LNet: Skipped 1 previous similar message [ 847.887771] LNet: 78651:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 847.895409] LNet: 78651:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 848.019653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 848.081913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 848.872296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp -m GET -e remote_dropped -r 2 [ 848.937160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.111@tcp1 -m GET -e remote_dropped -r 2 [ 848.985687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp -m GET -e remote_dropped -r 2 [ 849.041637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp -d 192.168.202.11@tcp1 -m GET -e remote_dropped -r 2 [ 849.101834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp -m GET -e remote_dropped -r 2 [ 849.149044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.111@tcp1 -m GET -e remote_dropped -r 2 [ 849.191171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp -m GET -e remote_dropped -r 2 [ 849.235872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.11@tcp1 -d 192.168.202.11@tcp1 -m GET -e remote_dropped -r 2 [ 849.303884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.11@tcp 192.168.202.111@tcp [ 849.370952] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.111@tcp [ 849.853915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.879943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 855.143467] LNet: 78828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.155413] LNetError: 78828:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.167588] LNet: Removed LNI 192.168.202.11@tcp [ 855.191746] LNet: Removed LNI 192.168.202.11@tcp1 [ 855.703310] Key type .llcrypt unregistered [ 855.704656] Key type ._llcrypt unregistered [ 874.801728] Key type ._llcrypt registered [ 874.803223] Key type .llcrypt registered [ 875.830682] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:48:18 (1756810098) [ 876.365539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.374235] alg: No test for adler32 (adler32-zlib) [ 877.242217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.280098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.305663] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 877.967581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 878.027126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 878.075642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 878.154297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 878.222314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 878.298537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 882.843228] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:48:25 (1756810105) [ 883.084894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.098954] LNet: 80381:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.102219] LNetError: 80381:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.118880] LNet: Removed LNI 192.168.202.11@tcp [ 883.138737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.181527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.203360] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 888.191209] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:48:30 (1756810110) [ 889.192472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 890.402541] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:48:33 (1756810113) [ 890.747656] LNet: 81412:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.755801] LNetError: 81412:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.767703] LNet: Removed LNI 192.168.202.11@tcp [ 891.156599] Key type .llcrypt unregistered [ 891.160058] Key type ._llcrypt unregistered [ 894.684202] Key type ._llcrypt registered [ 894.685859] Key type .llcrypt registered [ 894.853571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.868725] alg: No test for adler32 (adler32-zlib) [ 895.710333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 895.727376] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 900.191665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 903.039706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.111@tcp [ 904.291978] LNet: 81822:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.297957] LNetError: 81822:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.316176] LNet: Removed LNI 192.168.202.11@tcp [ 904.604404] Key type .llcrypt unregistered [ 904.605903] Key type ._llcrypt unregistered [ 910.133733] Key type ._llcrypt registered [ 910.138694] Key type .llcrypt registered [ 910.969588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 911.895302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 912.838277] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:48:55 (1756810135) [ 913.176092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.188840] alg: No test for adler32 (adler32-zlib) [ 914.041953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.110290] LNet: 82906:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.535370] Key type .llcrypt unregistered [ 914.537107] Key type ._llcrypt unregistered [ 917.770937] Key type ._llcrypt registered [ 917.772110] Key type .llcrypt registered [ 917.925284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.933196] alg: No test for adler32 (adler32-zlib) [ 918.828965] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 918.830663] LNet: 83158:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 918.849150] LNetError: 83160:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.13@tcp has gone from up to down [ 918.858040] LNetError: 83170:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.14@tcp is being used as a gateway but routing feature is not turned on [ 919.171404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.273400] LNetError: 83170:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.15@tcp has gone from up to down [ 922.281656] LNetError: 83170:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 922.287729] LNet: 83216:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.297288] LNetError: 83216:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.316795] LNet: Removed LNI 192.168.202.11@tcp [ 922.329543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.377618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 922.391345] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 922.398224] LNet: 83223:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 922.402808] LNet: 83223:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 922.407072] LNetError: 83223:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.12@tcp is a gateway. Can not delete it [ 922.413969] LNetError: 83223:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.202.12@tcp is a router. Can not be deleted [ 922.419140] LNet: 83223:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 922.441654] LNetError: 83220:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.202.16@tcp is being used as a gateway but routing feature is not turned on [ 922.457053] LNetError: 83220:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 925.028829] LNetError: 83234:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.202.15@tcp has gone from up to down [ 925.034709] LNetError: 83234:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 926.576670] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:49:09 (1756810149) [ 927.720311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg211-server [ 928.797555] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:49:11 (1756810151) [ 929.930387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg211-server [ 931.155876] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:49:14 (1756810154) [ 932.269916] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 934.056271] Key type lgssc registered [ 934.875573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.010065] Key type lgssc unregistered [ 948.276817] LNet: 85643:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.283687] LNetError: 85643:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.298692] LNet: Removed LNI 192.168.202.11@tcp [ 948.665997] Key type .llcrypt unregistered [ 948.668107] Key type ._llcrypt unregistered [ 957.194120] Key type ._llcrypt registered [ 957.195767] Key type .llcrypt registered [ 957.965475] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:49:40 (1756810180) [ 958.240750] Key type .llcrypt unregistered [ 958.243727] Key type ._llcrypt unregistered [ 961.250647] Key type ._llcrypt registered [ 961.252355] Key type .llcrypt registered [ 961.373677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.390248] alg: No test for adler32 (adler32-zlib) [ 962.260280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.316379] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 962.318635] LNet: *** cfs_fail_loc=e003, val=100*** [ 962.797573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 962.814662] Ping reply received with mapping: [ 962.816517] found metadata reply NI[192.168.202.11@tcp] [ 962.819844] 0b ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 962.823692] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 962.829574] 5a 5a 5a 5a 5a 5a 5a 5a [ 962.903133] LNet: 86785:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.910759] LNetError: 86785:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.919706] LNet: Removed LNI 192.168.202.11@tcp [ 963.307621] Key type .llcrypt unregistered [ 963.309805] Key type ._llcrypt unregistered [ 969.386656] Key type ._llcrypt registered [ 969.388322] Key type .llcrypt registered [ 970.256396] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:49:53 (1756810193) [ 970.523216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.533369] alg: No test for adler32 (adler32-zlib) [ 971.549678] LNetError: 87624:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 971.808550] Key type .llcrypt unregistered [ 971.810250] Key type ._llcrypt unregistered [ 975.224088] Key type ._llcrypt registered [ 975.226793] Key type .llcrypt registered [ 976.084847] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:49:58 (1756810198) [ 976.341374] Key type .llcrypt unregistered [ 976.344566] Key type ._llcrypt unregistered [ 979.372333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 979.381812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.650837] Key type ._llcrypt registered [ 981.652693] Key type .llcrypt registered [ 981.751590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.760222] alg: No test for adler32 (adler32-zlib) [ 982.631503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.672869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 987.556463] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:50:10 (1756810210) [ 987.602278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.615825] LNet: 89388:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.619920] LNetError: 89388:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 987.633703] LNet: Removed LNI 10.1.2.3@tcp [ 987.645318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.953607] Key type .llcrypt unregistered [ 987.955155] Key type ._llcrypt unregistered [ 990.201347] Key type ._llcrypt registered [ 990.202964] Key type .llcrypt registered [ 990.271471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.277907] alg: No test for adler32 (adler32-zlib) [ 991.094213] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 991.121947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 991.189840] LNet: 89676:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.192715] LNetError: 89676:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.200330] LNet: Removed LNI 192.168.202.11@tcp [ 991.471599] Key type .llcrypt unregistered [ 991.473312] Key type ._llcrypt unregistered [ 996.140666] Key type ._llcrypt registered [ 996.142161] Key type .llcrypt registered [ 996.801863] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:50:19 (1756810219) [ 997.108296] Key type .llcrypt unregistered [ 997.109971] Key type ._llcrypt unregistered [ 1000.188726] Key type ._llcrypt registered [ 1000.193158] Key type .llcrypt registered [ 1000.312575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.332918] alg: No test for adler32 (adler32-zlib) [ 1005.278473] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:50:28 (1756810228) [ 1005.408585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.441857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.490767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1005.508150] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1006.051874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1006.077677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1006.105521] LNet: Removed LNI 192.168.202.11@tcp [ 1006.312370] LNet: 91406:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.593931] Key type .llcrypt unregistered [ 1006.595465] Key type ._llcrypt unregistered [ 1013.256857] Key type ._llcrypt registered [ 1013.258674] Key type .llcrypt registered [ 1014.170536] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:50:37 (1756810237) [ 1014.391355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.402477] alg: No test for adler32 (adler32-zlib) [ 1015.271068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.301315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1015.315666] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1015.709302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 1015.729472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.11@tcp [ 1015.741845] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1015.743773] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1015.745241] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1015, 0, 0 [ 1015.747593] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1015.748877] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1015.751090] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1015.752802] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1015, 0, 0 [ 1015.755490] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1015.757414] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1015.759533] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1015.761058] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1015, 0, 1065 [ 1015.763780] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1015.765155] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1015.766819] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1015.768653] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1015, 0, 1065 [ 1015.771061] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1019.360735] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:50:42 (1756810242) [ 1019.642106] LNet: 92940:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.646326] LNetError: 92940:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.656570] LNet: Removed LNI 192.168.202.11@tcp [ 1020.153243] Key type .llcrypt unregistered [ 1020.155584] Key type ._llcrypt unregistered [ 1023.386414] Key type ._llcrypt registered [ 1023.388017] Key type .llcrypt registered [ 1023.518770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.531044] alg: No test for adler32 (adler32-zlib) [ 1024.372671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1024.390787] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1028.420660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 1032.546821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.568614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1032.580186] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 1033.168932] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.236549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.239059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.242272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.484894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1035.507373] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1042.221827] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:51:05 (1756810265) [ 1042.331556] LNet: 94429:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.342368] LNetError: 94429:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.347341] LNet: Removed LNI 192.168.202.11@tcp [ 1043.360186] LNet: Removed LNI 192.168.202.11@tcp1 [ 1043.363507] LNetError: 94429:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.371318] LNetError: 94429:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1043.384289] LNet: Removed LNI 10.1.2.3@tcp99 [ 1043.816529] Key type .llcrypt unregistered [ 1043.818156] Key type ._llcrypt unregistered [ 1047.079168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1047.082835] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.384160] Key type ._llcrypt registered [ 1049.385186] Key type .llcrypt registered [ 1049.507023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.517749] alg: No test for adler32 (adler32-zlib) [ 1050.390103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.451411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1050.476554] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1050.967111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1050.980316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1052.352124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1052.375683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1052.402702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1052.416231] LNetError: 95169:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1052.438747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1052.474748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 1052.778549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1052.831712] LNet: 95189:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.835512] LNetError: 95189:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.840079] LNet: Removed LNI 192.168.202.11@tcp [ 1052.845738] LNet: Removed LNI 10.1.2.3@tcp [ 1053.258794] Key type .llcrypt unregistered [ 1053.262076] Key type ._llcrypt unregistered [ 1060.087264] Key type ._llcrypt registered [ 1060.088818] Key type .llcrypt registered [ 1061.182555] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:51:23 (1756810283) [ 1061.477863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.492880] alg: No test for adler32 (adler32-zlib) [ 1062.375475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.425680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.448275] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1063.059844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.11@tcp [ 1063.133459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg211-client@tcp [ 1067.168209] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:51:30 (1756810290) [ 1067.241190] LNet: 96708:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.244657] LNetError: 96708:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.257410] LNet: Removed LNI 192.168.202.11@tcp [ 1067.631760] Key type .llcrypt unregistered [ 1067.634433] Key type ._llcrypt unregistered [ 1071.046054] Key type ._llcrypt registered [ 1071.047606] Key type .llcrypt registered [ 1071.163745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.175118] alg: No test for adler32 (adler32-zlib) [ 1072.035481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1072.053075] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1072.077728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1076.657262] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:51:39 (1756810299) [ 1077.563489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1078.648042] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:51:41 (1756810301) [ 1079.638314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1080.653846] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:51:43 (1756810303) [ 1080.809975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1080.839635] LNet: 98127:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.843853] LNetError: 98127:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.856186] LNet: Removed LNI 192.168.202.11@tcp [ 1080.871853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.908742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.925788] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1081.432920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1081.463164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1085.656122] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:51:48 (1756810308) [ 1085.771950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.790502] LNet: 98833:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.795115] LNetError: 98833:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.825827] LNet: Removed LNI 192.168.202.11@tcp [ 1086.849088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.881777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1086.902932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1087.075395] LNet: 98869:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.355500] Key type .llcrypt unregistered [ 1087.357118] Key type ._llcrypt unregistered [ 1094.208070] Key type ._llcrypt registered [ 1094.213640] Key type .llcrypt registered [ 1095.171587] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:51:57 (1756810317) [ 1095.493582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.507770] alg: No test for adler32 (adler32-zlib) [ 1096.359926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.403496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1096.421547] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1096.942419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1096.976326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 1100.870858] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:52:03 (1756810323) [ 1100.978213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.991489] LNet: 100382:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.995598] LNetError: 100382:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1101.003951] LNet: Removed LNI 192.168.202.11@tcp [ 1101.016351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.049057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1101.074436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1104.774502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1105.781715] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:52:08 (1756810328) [ 1105.935298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.958287] LNet: 101100:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.973818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.147482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1106.162602] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1106.606774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1106.621030] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.108919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.11@tcp [ 1107.150061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1107.187752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.11@tcp -m PUT -e local_timeout -r 1 [ 1107.237863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1107.286247] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1107.288267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1107.431805] LNetError: 101133:0:(lib-move.c:4945:lnet_parse()) 192.168.202.11@tcp, src 192.168.202.11@tcp: Dropping PUT (error -108 looking up sender) [ 1108.448868] LNet: 101408:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.460725] LNetError: 101408:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.466302] LNet: Removed LNI 192.168.202.11@tcp [ 1108.483163] LNet: Removed LNI 10.1.2.3@tcp [ 1108.681857] Key type .llcrypt unregistered [ 1108.683077] Key type ._llcrypt unregistered [ 1114.974775] Key type ._llcrypt registered [ 1114.976508] Key type .llcrypt registered [ 1115.837739] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:52:18 (1756810338) [ 1116.142409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1116.156159] alg: No test for adler32 (adler32-zlib) [ 1117.006703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.273105] LNet: 102257:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.698745] Key type .llcrypt unregistered [ 1117.703736] Key type ._llcrypt unregistered [ 1121.463388] Key type ._llcrypt registered [ 1121.465329] Key type .llcrypt registered [ 1121.642658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.664665] alg: No test for adler32 (adler32-zlib) [ 1122.558980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1122.606737] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1128.141358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 1128.342605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1128.836228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.111@tcp [ 1129.091489] LNet: 102693:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.158729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.181211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.628313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1129.824313] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.111@tcp [ 1129.856209] LNet: 102799:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.895167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.911543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.472214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.111@tcp [ 1130.905887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.931570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.587901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1131.920519] LNet: 103009:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.938076] LNet: 103009:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 1132.025733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.053277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.340263] LNet: 103050:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.349109] LNetError: 103050:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.369690] LNet: Removed LNI 192.168.202.11@tcp [ 1132.734253] Key type .llcrypt unregistered [ 1132.735676] Key type ._llcrypt unregistered [ 1142.928265] Key type ._llcrypt registered [ 1142.929829] Key type .llcrypt registered [ 1143.916857] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:52:46 (1756810366) [ 1144.374621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.383234] alg: No test for adler32 (adler32-zlib) [ 1145.270500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1145.544962] LNet: 103934:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.055549] Key type .llcrypt unregistered [ 1146.057977] Key type ._llcrypt unregistered [ 1149.900440] Key type ._llcrypt registered [ 1149.902307] Key type .llcrypt registered [ 1150.072887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.089414] alg: No test for adler32 (adler32-zlib) [ 1150.978897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1151.009530] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1156.396781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.111@tcp [ 1161.686510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.718583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1161.730153] LNet: Added LNI 192.168.202.11@tcp1 [8/256/0/180] [ 1162.410827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1162.745443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.111@tcp [ 1162.992157] LNet: 104569:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.039465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.066818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.450939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1163.739303] LNet: 104674:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.785872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.809794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.219962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.111@tcp [ 1164.330556] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.111@tcp [ 1164.519955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.560343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.036358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1165.272921] LNet: 104885:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.276418] LNet: 104885:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 1165.316714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.345938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.550589] LNet: 104960:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.556094] LNetError: 104960:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.564072] LNet: Removed LNI 192.168.202.11@tcp [ 1168.574276] LNet: Removed LNI 192.168.202.11@tcp1 [ 1168.845095] Key type .llcrypt unregistered [ 1168.847112] Key type ._llcrypt unregistered [ 1178.291864] Key type ._llcrypt registered [ 1178.293524] Key type .llcrypt registered [ 1179.039591] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1010 sec ========== 06:53:22 (1756810402) [ 1179.454534] Key type .llcrypt unregistered [ 1179.456450] Key type ._llcrypt unregistered [ 1187.660659] Key type ._llcrypt registered [ 1187.678066] Key type .llcrypt registered [ 1187.898417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1187.907634] alg: No test for adler32 (adler32-zlib) [ 1188.883804] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1189.038899] LNet: Added LNI 192.168.202.11@tcp [8/256/0/180] [ 1190.681365] Key type lgssc registered [ 1191.420674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.496564] Lustre: Mounted lustre-client [ 1233.197510] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1242.437994] LustreError: 108177:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff98e98a9e9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1242.453864] LustreError: 108177:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1242.478769] Lustre: Unmounted lustre-client [ 1256.998361] Key type lgssc unregistered [ 1257.247198] LNet: 108657:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.251960] LNetError: 108657:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.275278] LNet: Removed LNI 192.168.202.11@tcp [ 1257.858186] Key type .llcrypt unregistered [ 1257.863733] Key type ._llcrypt unregistered