[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 437990618 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003243] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007586] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.011104] LSM: Security Framework initializing [ 0.012032] Yama: becoming mindful. [ 0.013023] SELinux: Initializing. [ 0.014044] *** VALIDATE selinux *** [ 0.021595] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026110] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028073] *** VALIDATE tmpfs *** [ 0.029378] *** VALIDATE proc *** [ 0.030190] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.038883] debug: unmapping init [mem 0xffffffffb1a59000-0xffffffffb1a60fff] [ 0.040158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041609] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042016] ... version: 2 [ 0.043008] ... bit width: 48 [ 0.044006] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.046007] ... max period: 00007fffffffffff [ 0.047005] ... fixed-purpose events: 3 [ 0.047718] ... event mask: 000000070000000f [ 0.048277] rcu: Hierarchical SRCU implementation. [ 0.050388] smp: Bringing up secondary CPUs ... [ 0.051360] x86: Booting SMP configuration: [ 0.052013] .... node #0, CPUs: #1 #2 #3 [ 0.054778] smp: Brought up 1 node, 4 CPUs [ 0.055918] smpboot: Max logical packages: 1 [ 0.056008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.229065] node 0 deferred pages initialised in 172ms [ 0.231359] devtmpfs: initialized [ 0.232143] x86/mm: Memory block size: 128MB [ 0.234551] gcov: version magic: 0x41383552 [ 0.235501] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.236082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.237247] pinctrl core: initialized pinctrl subsystem [ 0.238160] [ 0.238685] ************************************************************* [ 0.239011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240007] ** ** [ 0.241009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242010] ** ** [ 0.243006] ** This means that this kernel is built to expose internal ** [ 0.244007] ** IOMMU data structures, which may compromise security on ** [ 0.245008] ** your system. ** [ 0.246010] ** ** [ 0.247007] ** If you see this message and you are not debugging the ** [ 0.248007] ** kernel, report this immediately to your vendor! ** [ 0.249006] ** ** [ 0.250006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251005] ************************************************************* [ 0.252486] NET: Registered protocol family 16 [ 0.253299] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255023] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.256267] cpuidle: using governor menu [ 0.257136] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.259440] PCI: Using configuration type 1 for base access [ 0.261095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.266071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268102] cryptd: max_cpu_qlen set to 1000 [ 0.269830] ACPI: Added _OSI(Module Device) [ 0.272013] ACPI: Added _OSI(Processor Device) [ 0.273007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.274006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278449] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.283324] ACPI: Interpreter enabled [ 0.284035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.284879] ACPI: Using IOAPIC for interrupt routing [ 0.286050] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.287214] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.293158] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.294016] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.296007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.299056] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.301810] acpiphp: Slot [2] registered [ 0.303090] acpiphp: Slot [3] registered [ 0.303830] acpiphp: Slot [4] registered [ 0.304038] acpiphp: Slot [5] registered [ 0.304817] acpiphp: Slot [6] registered [ 0.306087] acpiphp: Slot [7] registered [ 0.308062] acpiphp: Slot [8] registered [ 0.309074] acpiphp: Slot [9] registered [ 0.311073] acpiphp: Slot [10] registered [ 0.312062] acpiphp: Slot [11] registered [ 0.313047] acpiphp: Slot [12] registered [ 0.314062] acpiphp: Slot [13] registered [ 0.316091] acpiphp: Slot [14] registered [ 0.317048] acpiphp: Slot [15] registered [ 0.317812] acpiphp: Slot [16] registered [ 0.318038] acpiphp: Slot [17] registered [ 0.318836] acpiphp: Slot [18] registered [ 0.320040] acpiphp: Slot [19] registered [ 0.320893] acpiphp: Slot [20] registered [ 0.321037] acpiphp: Slot [21] registered [ 0.321782] acpiphp: Slot [22] registered [ 0.323037] acpiphp: Slot [23] registered [ 0.323824] acpiphp: Slot [24] registered [ 0.324036] acpiphp: Slot [25] registered [ 0.324750] acpiphp: Slot [26] registered [ 0.326054] acpiphp: Slot [27] registered [ 0.326795] acpiphp: Slot [28] registered [ 0.327036] acpiphp: Slot [29] registered [ 0.327765] acpiphp: Slot [30] registered [ 0.329038] acpiphp: Slot [31] registered [ 0.329794] PCI host bridge to bus 0000:00 [ 0.330011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336008] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.338014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342607] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345366] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348611] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.351260] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358438] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361374] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.364007] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371009] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.373764] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.376677] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.405009] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.413241] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418997] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437014] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.444113] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.445177] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.447162] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.448160] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.449115] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.453007] iommu: Default domain type: Passthrough [ 0.454492] SCSI subsystem initialized [ 0.456098] ACPI: bus type USB registered [ 0.457088] usbcore: registered new interface driver usbfs [ 0.458045] usbcore: registered new interface driver hub [ 0.460068] usbcore: registered new device driver usb [ 0.461167] pps_core: LinuxPPS API ver. 1 registered [ 0.463007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.466033] PTP clock support registered [ 0.467170] EDAC MC: Ver: 3.0.0 [ 0.469117] PCI: Using ACPI for IRQ routing [ 0.470676] NetLabel: Initializing [ 0.471007] NetLabel: domain hash size = 128 [ 0.473006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.474052] NetLabel: unlabeled traffic allowed by default [ 0.476085] vgaarb: loaded [ 0.477256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.479009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.484316] clocksource: Switched to clocksource kvm-clock [ 0.595129] VFS: Disk quotas dquot_6.6.0 [ 0.596463] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.598671] *** VALIDATE ramfs *** [ 0.599770] *** VALIDATE hugetlbfs *** [ 0.601120] pnp: PnP ACPI init [ 0.603875] pnp: PnP ACPI: found 6 devices [ 0.618336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.621329] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.623253] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.625492] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.627977] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.630368] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.632713] NET: Registered protocol family 2 [ 0.635278] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.639706] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.642748] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.647508] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.650949] TCP: Hash tables configured (established 65536 bind 65536) [ 0.653525] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.655437] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657348] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.659718] NET: Registered protocol family 1 [ 0.661687] RPC: Registered named UNIX socket transport module. [ 0.663286] RPC: Registered udp transport module. [ 0.665184] RPC: Registered tcp transport module. [ 0.667018] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.669141] NET: Registered protocol family 44 [ 0.670326] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.671678] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.673621] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.675586] PCI: CLS 0 bytes, default 64 [ 0.676828] Unpacking initramfs... [ 2.233584] debug: unmapping init [mem 0xffff97017cc64000-0xffff97017ffcffff] [ 2.238280] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.240762] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.244491] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.842843] Initialise system trusted keyrings [ 2.845205] Key type blacklist registered [ 2.850576] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.867954] zbud: loaded [ 2.875572] *** VALIDATE nfs *** [ 2.877139] *** VALIDATE nfs4 *** [ 2.878916] pstore: using deflate compression [ 2.882172] Platform Keyring initialized [ 3.043673] NET: Registered protocol family 38 [ 3.046238] Key type asymmetric registered [ 3.047450] Asymmetric key parser 'x509' registered [ 3.050158] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.056537] io scheduler mq-deadline registered [ 3.058654] io scheduler kyber registered [ 3.060330] io scheduler bfq registered [ 3.065158] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.072430] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.075132] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.089505] ACPI: Power Button [PWRF] [ 3.245140] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.370131] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.486810] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.529987] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.559477] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.575952] Non-volatile memory driver v1.3 [ 3.577451] Linux agpgart interface v0.103 [ 3.636262] virtio_blk virtio1: [vda] 133256 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.643520] vda: detected capacity change from 0 to 68227072 [ 3.693266] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.699964] vdb: detected capacity change from 0 to 1073741824 [ 3.713453] libphy: Fixed MDIO Bus: probed [ 3.731301] usbcore: registered new interface driver usbserial_generic [ 3.735098] usbserial: USB Serial support registered for generic [ 3.738484] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.748032] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.750050] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.752739] mousedev: PS/2 mouse device common for all mice [ 3.757450] rtc_cmos 00:05: RTC can wake from S4 [ 3.762769] rtc_cmos 00:05: registered as rtc0 [ 3.764663] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.767387] intel_pstate: CPU model not supported [ 3.768158] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.776799] hid: raw HID events driver (C) Jiri Kosina [ 3.778149] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.787623] usbcore: registered new interface driver usbhid [ 3.792389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.795027] usbhid: USB HID core driver [ 3.798119] drop_monitor: Initializing network drop monitor service [ 3.806773] Initializing XFRM netlink socket [ 3.809035] NET: Registered protocol family 10 [ 3.814143] Segment Routing with IPv6 [ 3.816616] NET: Registered protocol family 17 [ 3.819812] mpls_gso: MPLS GSO support [ 3.832304] RAS: Correctable Errors collector initialized. [ 3.834331] AVX version of gcm_enc/dec engaged. [ 3.837962] AES CTR mode by8 optimization enabled [ 3.994377] sched_clock: Marking stable (3994312734, 0)->(4806710659, -812397925) [ 3.997874] registered taskstats version 1 [ 3.999718] Loading compiled-in X.509 certificates [ 4.001956] zswap: loaded using pool lzo/zbud [ 4.048274] Key type big_key registered [ 4.067162] Key type encrypted registered [ 4.068748] ima: No TPM chip found, activating TPM-bypass! [ 4.070774] ima: Allocated hash algorithm: sha1 [ 4.072580] ima: No architecture policies found [ 4.074613] evm: Initialising EVM extended attributes: [ 4.076671] evm: security.selinux [ 4.078139] evm: security.ima [ 4.079267] evm: security.capability [ 4.080543] evm: HMAC attrs: 0x1 [ 4.083824] rtc_cmos 00:05: setting system clock to 2025-07-14 16:14:58 UTC (1752509698) [ 4.090563] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 4.094739] debug: unmapping init [mem 0xffffffffb1782000-0xffffffffb1a58fff] [ 4.102910] Write protecting the kernel read-only data: 28672k [ 4.107419] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 4.112217] debug: unmapping init [mem 0xffffffffb0714000-0xffffffffb07fffff] [ 4.158370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.169301] systemd[1]: Detected virtualization kvm. [ 4.172332] systemd[1]: Detected architecture x86-64. [ 4.175345] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.204252] systemd[1]: No hostname configured. [ 4.207220] systemd[1]: Set hostname to . [ 4.210221] random: systemd: uninitialized urandom read (16 bytes read) [ 4.227234] systemd[1]: Initializing machine ID from random generator. [ 4.419311] random: systemd: uninitialized urandom read (16 bytes read) [ 4.422174] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.427996] random: systemd: uninitialized urandom read (16 bytes read) [ 4.430491] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.434558] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.677187] device-mapper: uevent: version 1.0.3 [ 5.686378] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.882117] virtio_net virtio0 ens2: renamed from eth0 [ 7.350473] scsi host0: ata_piix [ 7.434693] scsi host1: ata_piix [ 7.436593] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.442501] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.623173] random: fast init done [ 12.611029] random: crng init done [ 12.613367] random: 7 urandom warning(s) missed due to ratelimiting [ 12.689390] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.234740] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.721761] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.270782] SELinux: Disabled at runtime. [ 17.344641] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.358487] systemd[1]: Detected virtualization kvm. [ 17.360078] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.595681] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.599958] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.606898] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.614382] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.620489] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.637833] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.654599] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 18.863513] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.903793] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.576252] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.669446] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.923469] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.004866] EDAC sbridge: Ver: 1.1.2 [ 23.914943] Key type dns_resolver registered [ 24.357476] NFS: Registering the id_resolver key type [ 24.359596] Key type id_resolver registered [ 24.361106] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg424-client login: [ 68.636504] libcfs: loading out-of-tree module taints kernel. [ 68.659608] Key type ._llcrypt registered [ 68.661320] Key type .llcrypt registered [ 68.999222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.008057] alg: No test for adler32 (adler32-zlib) [ 70.014173] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 70.365772] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 70.368976] LNet: Accept secure, port 988 [ 71.991386] Key type lgssc registered [ 72.578873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.741694] Lustre: Mounted lustre-client [ 128.408530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 142.212172] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing check_logdir /tmp/testlogs/ [ 143.813516] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing yml_node [ 146.069458] Lustre: DEBUG MARKER: Client: 2.16.57.2 [ 147.130119] Lustre: DEBUG MARKER: MDS: 2.16.57.2 [ 148.175570] Lustre: DEBUG MARKER: OSS: 2.16.57.2 [ 148.833403] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 14 12:17:22 EDT 2025 [ 151.519222] Lustre: lustre-OST0000-osc-ffff9701d0770000: disconnect after 24s idle [ 155.037604] LustreError: 7920:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9701d0770000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 155.042429] LustreError: 7920:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 155.055098] Lustre: Unmounted lustre-client [ 180.691275] Key type lgssc unregistered [ 180.812436] LNet: 8368:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 181.861797] LNet: Removed LNI 192.168.204.24@tcp [ 182.158059] Key type .llcrypt unregistered [ 182.159446] Key type ._llcrypt unregistered [ 191.449316] Key type ._llcrypt registered [ 191.450526] Key type .llcrypt registered [ 191.524700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 191.530667] alg: No test for adler32 (adler32-zlib) [ 192.331538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 192.354368] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 192.355936] LNet: Accept secure, port 988 [ 192.363700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 193.685193] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 193.713445] LNet: 9232:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.720313] LNet: Removed LNI 192.168.204.24@tcp [ 193.976448] Key type .llcrypt unregistered [ 193.977877] Key type ._llcrypt unregistered [ 196.053755] Key type ._llcrypt registered [ 196.054899] Key type .llcrypt registered [ 196.557404] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:18:10 (1752509890) [ 196.684396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 196.691430] alg: No test for adler32 (adler32-zlib) [ 197.505209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.541767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.549939] LNet: 9645:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 197.559688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.584369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 197.593596] LNet: 9652:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 200.035563] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:18:13 (1752509893) [ 200.072330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.083199] LNet: 10222:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 200.094631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.129654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 202.806093] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:18:16 (1752509896) [ 202.845596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.858982] LNet: 10807:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.874366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.919562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 205.358217] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:18:19 (1752509899) [ 205.386326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.397594] LNet: 11390:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 205.408457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.451807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 207.843588] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:18:21 (1752509901) [ 207.882118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.900319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.944755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 208.098294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 210.353742] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:18:24 (1752509904) [ 210.393621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.402620] LNet: 12566:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 210.404813] LNet: 12566:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 210.416794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.446101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 210.593579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 210.734917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 212.916397] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:18:26 (1752509906) [ 212.950318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.970275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.010153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 215.435226] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:18:29 (1752509909) [ 215.472129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.497125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 215.535400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 215.556197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 215.737376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 215.756477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 215.938513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 215.955321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 216.117821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 216.135898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 216.288477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 216.305200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 216.460522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 216.475603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 216.642862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 216.658936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 216.892740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 216.908817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 217.053971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 217.070047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 217.238612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 217.256290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 217.428100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 217.445816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 217.609655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 217.633947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 217.805204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 217.820517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 220.837893] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:18:34 (1752509914) [ 220.895655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.908274] LNet: 14426:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 220.911637] LNet: 14426:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 220.930373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.969541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 220.986623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 223.836604] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:18:37 (1752509917) [ 223.885205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.908576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.959447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 223.978930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 226.569621] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:18:40 (1752509920) [ 226.601309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.618754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.663166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 226.682444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 229.473145] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:18:43 (1752509923) [ 229.521254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.541956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.600610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 229.622684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 232.140274] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:18:46 (1752509926) [ 232.176162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.194152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.228900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 232.243112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 234.490460] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:18:48 (1752509928) [ 234.532678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.553856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.597795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 234.611711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 236.635851] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:18:50 (1752509930) [ 236.668460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.685681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.729778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.864858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 238.878792] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:18:52 (1752509932) [ 238.906860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.915568] LNet: 18531:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.917354] LNet: 18531:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 238.925971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.999556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.073378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 241.221547] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:18:55 (1752509935) [ 241.251582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.269489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.321343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 241.405298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 243.712239] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:18:57 (1752509937) [ 243.754769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.774026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.931741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.032699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 246.141417] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:19:00 (1752509940) [ 246.173851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.190570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.226196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.258656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 248.465343] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:19:02 (1752509942) [ 248.512033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 248.534977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.566728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 248.583902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 250.909918] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:19:04 (1752509944) [ 250.944199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.961158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.994312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 253.353192] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:19:07 (1752509947) [ 253.394747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.412637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.459890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 255.660109] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:19:09 (1752509949) [ 255.689346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.708436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.743223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 255.896581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 258.067292] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:19:12 (1752509952) [ 258.100496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.115832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.151214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 258.164440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 260.388898] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:19:14 (1752509954) [ 260.423418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.440903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.471725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 260.485109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 262.643895] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:19:16 (1752509956) [ 262.673228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.692234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.727743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 262.743172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 265.007958] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:19:18 (1752509958) [ 265.043604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.066732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.109285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 265.130336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 267.288420] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:19:21 (1752509961) [ 267.314706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.333178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.354466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 267.374914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 267.383316] LNetError: 25730:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 267.399153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 267.410252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 269.433509] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:19:23 (1752509963) [ 269.470690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.488076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.516180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.I2AKq1YEAb [ 269.539578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.I2AKq1YEAb [ 269.562300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.I2AKq1YEAb [ 271.601194] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:19:25 (1752509965) [ 271.634037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.643919] LNet: 26897:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.646136] LNet: 26897:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 271.656632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.676242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 271.689495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 271.709903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 274.032739] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:19:27 (1752509967) [ 274.063371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.079645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.104977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 274.121045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 274.136586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 274.152722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 274.167141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 274.179782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 274.192111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 274.204558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 274.218341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 274.231450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 274.248213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 274.268988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 274.296546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 274.320629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 274.340879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 274.356481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 274.374717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 274.391402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 274.409186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 274.419843] LNetError: 27529:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 276.534606] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:19:30 (1752509970) [ 276.585514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.603356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.632216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 279.147472] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:19:33 (1752509973) [ 279.838177] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg424-server [ 280.454079] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:19:34 (1752509974) [ 280.490768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.512686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.542344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 280.556640] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 280.559410] LNet: Accept secure, port 988 [ 281.152423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[25-29]@tcp [ 281.162880] LNet: 29240:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 281.180449] LNetError: 29033:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 281.191164] LNetError: 29033:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.28@tcp has gone from up to down [ 283.436349] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:19:37 (1752509977) [ 283.453578] LNetError: 29823:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 284.126586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg424-server [ 284.255390] LNetError: 29039:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.27@tcp has gone from up to down [ 284.262883] LNetError: 29039:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 284.732900] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:19:38 (1752509978) [ 284.764558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.797625] LNet: Removed LNI 192.168.204.24@tcp [ 285.805244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.825749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 285.833042] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 285.835182] LNet: Accept secure, port 988 [ 286.051590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[25-30/2]@tcp [ 286.059945] LNet: 30288:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 286.064536] LNet: 30288:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 286.070929] LNetError: 30178:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 286.074091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[25-30/2]@tcp [ 286.075299] LNetError: 30178:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 286.081600] LNetError: 30178:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.29@tcp has gone from up to down [ 286.084166] LNetError: 30178:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 288.187463] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:19:42 (1752509982) [ 288.217920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.253393] LNet: Removed LNI 192.168.204.24@tcp [ 289.261693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.294706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 289.318973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.333718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.360918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.388432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.405466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.427235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 289.457498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.474662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.501879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.533576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.550300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.573480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 289.603930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.621651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.649150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.683673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.699865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.718862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 289.743276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.756185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.782217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.813767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.829322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.850561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 289.875439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.889312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.917799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.949465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.964387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.986466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 290.009039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.022655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.051542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 292.493593] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:19:46 (1752509986) [ 293.193864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg424-server [ 293.787286] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:19:47 (1752509987) [ 294.482423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg424-server [ 295.043743] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:19:48 (1752509988) [ 295.359671] Key type .llcrypt unregistered [ 295.361540] Key type ._llcrypt unregistered [ 297.401916] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 297.445843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 297.447988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 297.450694] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 297.589913] Key type ._llcrypt registered [ 297.591375] Key type .llcrypt registered [ 297.662306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 297.668810] alg: No test for adler32 (adler32-zlib) [ 298.478347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.502491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 298.513245] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 298.514870] LNet: Accept secure, port 988 [ 298.748369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 298.756249] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 299.497633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 299.507500] LNet: Removed LNI 10.1.2.3@tcp [ 299.548240] LNet: 32966:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.556349] LNet: Removed LNI 192.168.204.24@tcp [ 299.802478] Key type .llcrypt unregistered [ 299.803757] Key type ._llcrypt unregistered [ 303.778553] Key type ._llcrypt registered [ 303.779604] Key type .llcrypt registered [ 304.304785] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:19:58 (1752509998) [ 304.501371] Key type .llcrypt unregistered [ 304.503214] Key type ._llcrypt unregistered [ 306.527262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 306.592591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 306.595086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 306.598097] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 306.717081] Key type ._llcrypt registered [ 306.718741] Key type .llcrypt registered [ 306.793262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.800340] alg: No test for adler32 (adler32-zlib) [ 307.620092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.645592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 307.658398] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 307.660301] LNet: Accept secure, port 988 [ 307.928545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 307.939333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.860107] LNet: 34420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.862516] LNet: Removed LNI 192.168.204.24@tcp [ 308.867326] LNet: Removed LNI 10.1.2.3@tcp [ 309.136934] Key type .llcrypt unregistered [ 309.138131] Key type ._llcrypt unregistered [ 313.245393] Key type ._llcrypt registered [ 313.247115] Key type .llcrypt registered [ 313.827460] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:20:07 (1752510007) [ 314.035278] Key type .llcrypt unregistered [ 314.036994] Key type ._llcrypt unregistered [ 315.963657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 316.026869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 316.032902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 316.035358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 316.134787] Key type ._llcrypt registered [ 316.135951] Key type .llcrypt registered [ 316.201161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 316.207106] alg: No test for adler32 (adler32-zlib) [ 317.015752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.041513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 317.052718] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 317.054244] LNet: Accept secure, port 988 [ 317.706983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 317.715252] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 318.459511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 318.467351] LNet: Removed LNI 10.1.2.3@tcp [ 318.473928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 318.490444] LNet: Removed LNI 10.1.2.31@tcp [ 318.498090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 318.506927] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 318.751738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 318.762493] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 319.123095] LNet: 36295:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.125965] LNet: Removed LNI 10.1.2.3@tcp [ 319.132320] LNet: Removed LNI 10.1.2.31@tcp [ 319.379467] Key type .llcrypt unregistered [ 319.380697] Key type ._llcrypt unregistered [ 323.160592] Key type ._llcrypt registered [ 323.161915] Key type .llcrypt registered [ 323.642378] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:20:17 (1752510017) [ 323.776844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 323.783402] alg: No test for adler32 (adler32-zlib) [ 324.595146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.617914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 324.628874] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 324.630672] LNet: Accept secure, port 988 [ 324.930273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.943037] LNet: 37239:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.960351] LNet: Removed LNI 192.168.204.24@tcp [ 324.969758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.993356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.24@tcp [ 325.009570] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 325.014633] LNet: Accept secure, port 988 [ 325.309186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.24@tcp [ 325.329653] LNet: Removed LNI 192.168.204.24@tcp [ 325.637585] Key type .llcrypt unregistered [ 325.639293] Key type ._llcrypt unregistered [ 329.604952] Key type ._llcrypt registered [ 329.605927] Key type .llcrypt registered [ 330.093268] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:20:23 (1752510023) [ 330.791839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg424-server [ 331.426194] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:20:25 (1752510025) [ 331.642264] Key type .llcrypt unregistered [ 331.643688] Key type ._llcrypt unregistered [ 333.657706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 333.713939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 333.716138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 333.718596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 333.867961] Key type ._llcrypt registered [ 333.869578] Key type .llcrypt registered [ 333.937664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 333.943465] alg: No test for adler32 (adler32-zlib) [ 334.744900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.771293] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 334.773521] LNet: Accept secure, port 988 [ 335.024941] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 335.810624] LNet: 39170:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.812812] LNet: Removed LNI 192.168.204.24@tcp1 [ 335.820376] LNet: Removed LNI 10.1.2.3@tcp2 [ 336.064921] Key type .llcrypt unregistered [ 336.066425] Key type ._llcrypt unregistered [ 340.159564] Key type ._llcrypt registered [ 340.162526] Key type .llcrypt registered [ 340.663786] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:20:34 (1752510034) [ 340.787083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 340.792332] alg: No test for adler32 (adler32-zlib) [ 341.602308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.625191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 341.634990] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 341.637673] LNet: Accept secure, port 988 [ 341.907155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.24 [ 341.917802] LNet: Removed LNI 192.168.204.24@tcp [ 344.350357] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:20:38 (1752510038) [ 344.383733] LNet: 40688:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.393735] LNet: Removed LNI 192.168.204.24@tcp [ 344.647520] Key type .llcrypt unregistered [ 344.649083] Key type ._llcrypt unregistered [ 346.598102] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 346.706981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 346.708910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 346.711239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 346.849785] Key type ._llcrypt registered [ 346.851090] Key type .llcrypt registered [ 346.914088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 346.921632] alg: No test for adler32 (adler32-zlib) [ 347.763590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 347.765607] LNet: Accept secure, port 988 [ 350.434751] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:20:44 (1752510044) [ 350.483545] LNet: 41553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.491553] LNet: Removed LNI 10.1.2.3@tcp [ 350.828954] Key type .llcrypt unregistered [ 350.834316] Key type ._llcrypt unregistered [ 353.090194] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 353.193619] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 353.195498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 353.197632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 353.339319] Key type ._llcrypt registered [ 353.340691] Key type .llcrypt registered [ 353.413192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 353.422543] alg: No test for adler32 (adler32-zlib) [ 354.262768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 354.265283] LNet: Accept secure, port 988 [ 356.856994] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:20:50 (1752510050) [ 356.891279] LNet: 42419:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.898740] LNet: Removed LNI 10.1.2.3@tcp [ 357.163760] Key type .llcrypt unregistered [ 357.165397] Key type ._llcrypt unregistered [ 359.184501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 359.290556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 359.292573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 359.294568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 359.421901] Key type ._llcrypt registered [ 359.423709] Key type .llcrypt registered [ 359.477852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 359.483665] alg: No test for adler32 (adler32-zlib) [ 360.304429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 360.306406] LNet: Accept secure, port 988 [ 362.479185] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:20:56 (1752510056) [ 362.516313] LNet: 43284:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.525713] LNet: Removed LNI 10.1.2.3@tcp [ 362.753483] Key type .llcrypt unregistered [ 362.754545] Key type ._llcrypt unregistered [ 364.555681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 364.653593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 364.655437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 364.656773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 364.764764] Key type ._llcrypt registered [ 364.766142] Key type .llcrypt registered [ 364.825096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 364.830759] alg: No test for adler32 (adler32-zlib) [ 365.631685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.666665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 365.668286] LNet: Accept secure, port 988 [ 368.671362] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:21:02 (1752510062) [ 368.838107] LNet: 44488:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.851618] LNet: Removed LNI 10.1.2.3@tcp [ 369.104612] Key type .llcrypt unregistered [ 369.105801] Key type ._llcrypt unregistered [ 371.002063] Key type ._llcrypt registered [ 371.003495] Key type .llcrypt registered [ 371.067845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 371.075297] alg: No test for adler32 (adler32-zlib) [ 371.876323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 371.887200] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 371.888883] LNet: Accept secure, port 988 [ 374.701407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 374.898741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_interrupt [ 374.914408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_interrupt [ 374.929885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 374.939276] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 375.026459] LNet: 44901:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 375.051674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.064275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.078866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.226950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_dropped [ 375.240501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_dropped [ 375.255331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 375.375033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.389900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.402380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.552416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_aborted [ 375.564918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_aborted [ 375.577935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 375.584880] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 375.587474] LNet: Skipped 1 previous similar message [ 375.660068] LNet: 45071:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 375.662650] LNet: 45071:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 375.687557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.701730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.715475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.861988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_no_route [ 375.876675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_no_route [ 375.891803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 376.006328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.017386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.027575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 376.177911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_timeout [ 376.192709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_timeout [ 376.206445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 376.327622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.339454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.351471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 376.497184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_error [ 376.507918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_error [ 376.519563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 376.624653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.635568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.646749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 376.738214] LNet: 45362:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.765310] LNet: Removed LNI 192.168.204.24@tcp [ 377.915385] Key type .llcrypt unregistered [ 377.916460] Key type ._llcrypt unregistered [ 382.734904] Key type ._llcrypt registered [ 382.735935] Key type .llcrypt registered [ 383.201882] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:21:17 (1752510077) [ 383.488518] Key type .llcrypt unregistered [ 383.489620] Key type ._llcrypt unregistered [ 385.231221] Key type ._llcrypt registered [ 385.233421] Key type .llcrypt registered [ 385.293347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 385.298446] alg: No test for adler32 (adler32-zlib) [ 386.097768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 386.107925] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 386.109621] LNet: Accept secure, port 988 [ 388.946596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 392.158197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.172414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 392.181544] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 392.592904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_interrupt [ 392.606628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_interrupt [ 392.619169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_interrupt [ 392.633177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_interrupt [ 392.649678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_interrupt [ 392.664942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_interrupt [ 392.678777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_interrupt [ 392.692579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_interrupt [ 392.704919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 392.711630] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 392.713566] LNet: 1 local NIs in recovery (showing 1): 192.168.204.24@tcp [ 392.800437] LNet: 46795:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 392.823148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 392.835584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 392.848692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 393.007211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_dropped [ 393.022901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_dropped [ 393.037825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_dropped [ 393.051085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_dropped [ 393.063923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_dropped [ 393.077919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_dropped [ 393.089800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_dropped [ 393.102203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_dropped [ 393.118201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 393.236075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 393.249670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 393.261675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 393.409099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_aborted [ 393.423186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_aborted [ 393.438150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_aborted [ 393.453262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_aborted [ 393.465856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_aborted [ 393.480610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_aborted [ 393.495275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_aborted [ 393.507733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_aborted [ 393.519928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 393.528353] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 393.531806] LNet: Skipped 6 previous similar messages [ 393.608291] LNet: 46989:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 393.611504] LNet: 46989:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 393.637359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 393.649399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 393.663086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 393.812842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_no_route [ 393.827792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_no_route [ 393.847478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_no_route [ 393.862798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_no_route [ 393.877742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_no_route [ 393.896238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_no_route [ 393.924612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_no_route [ 393.952819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_no_route [ 393.972828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 394.092392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 394.105290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 394.117828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 394.275862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_timeout [ 394.291139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_timeout [ 394.303727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_timeout [ 394.318830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_timeout [ 394.334733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_timeout [ 394.350089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_timeout [ 394.364538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_timeout [ 394.380621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_timeout [ 394.397180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 394.522532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 394.537182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 394.549486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 394.709590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e local_error [ 394.723896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e local_error [ 394.739136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e local_error [ 394.754014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e local_error [ 394.766919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e local_error [ 394.782131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e local_error [ 394.797573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e local_error [ 394.814440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e local_error [ 394.830610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 394.842603] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 394.847618] LNet: Skipped 9 previous similar messages [ 394.934425] LNet: 47280:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 394.938405] LNet: 47280:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 394.966089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 394.980871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 394.995716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 396.039595] LNet: 47351:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.087113] LNet: Removed LNI 192.168.204.24@tcp [ 398.117547] LNet: Removed LNI 192.168.204.24@tcp1 [ 398.272774] Key type .llcrypt unregistered [ 398.274365] Key type ._llcrypt unregistered [ 403.432204] Key type ._llcrypt registered [ 403.433145] Key type .llcrypt registered [ 403.881520] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:21:37 (1752510097) [ 404.124217] Key type .llcrypt unregistered [ 404.125381] Key type ._llcrypt unregistered [ 405.984640] Key type ._llcrypt registered [ 405.985586] Key type .llcrypt registered [ 406.050925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.056913] alg: No test for adler32 (adler32-zlib) [ 406.853192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 406.861795] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 406.863508] LNet: Accept secure, port 988 [ 409.598547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 409.805439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_dropped [ 409.819363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_dropped [ 409.831603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 409.838380] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 409.932676] LNet: 48576:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 409.956550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 409.966748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 409.977796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 410.151879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_error [ 410.173806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_error [ 410.187607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 410.314456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 410.324591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 410.334228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 410.480337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_timeout [ 410.491884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_timeout [ 410.503956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 410.510937] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 410.513927] LNet: Skipped 1 previous similar message [ 410.594247] LNet: 48746:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 410.596673] LNet: 48746:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 410.620932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 410.632369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 410.643866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 410.724205] LNet: 48783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.734332] LNet: Removed LNI 192.168.204.24@tcp [ 410.897421] Key type .llcrypt unregistered [ 410.898357] Key type ._llcrypt unregistered [ 416.367629] Key type ._llcrypt registered [ 416.368992] Key type .llcrypt registered [ 416.787386] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:21:50 (1752510110) [ 417.023238] Key type .llcrypt unregistered [ 417.024849] Key type ._llcrypt unregistered [ 418.920049] Key type ._llcrypt registered [ 418.921537] Key type .llcrypt registered [ 418.994100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 419.000941] alg: No test for adler32 (adler32-zlib) [ 419.796739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 419.805522] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 419.807484] LNet: Accept secure, port 988 [ 422.599215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 425.086462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.097629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 425.105230] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 425.504821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_dropped [ 425.514884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_dropped [ 425.525036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_dropped [ 425.535688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_dropped [ 425.547274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e remote_dropped [ 425.558566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_dropped [ 425.570887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e remote_dropped [ 425.584141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_dropped [ 425.596184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 425.603070] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 425.605034] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.124@tcp [ 425.679152] LNet: 50215:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 425.703661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.715650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.727448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 425.738552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.900562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_error [ 425.915520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_error [ 425.931226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_error [ 425.943381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_error [ 425.955850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e remote_error [ 425.968734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_error [ 425.980970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e remote_error [ 425.995612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_error [ 426.012683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 426.120034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 426.134258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 426.149303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 426.161363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 426.319540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e remote_timeout [ 426.331720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_timeout [ 426.346188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e remote_timeout [ 426.359547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_timeout [ 426.374399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e remote_timeout [ 426.389440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e remote_timeout [ 426.404674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e remote_timeout [ 426.420232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e remote_timeout [ 426.434503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.124@tcp [ 426.441794] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 426.448203] LNet: Skipped 3 previous similar messages [ 426.537819] LNet: 50414:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 426.540731] LNet: 50414:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 426.564572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 426.575748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 426.586717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 426.597624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 428.652491] LNet: 50486:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.663133] LNet: Removed LNI 192.168.204.24@tcp [ 429.670290] LNet: Removed LNI 192.168.204.24@tcp1 [ 429.798844] Key type .llcrypt unregistered [ 429.799744] Key type ._llcrypt unregistered [ 434.724078] Key type ._llcrypt registered [ 434.725232] Key type .llcrypt registered [ 435.183042] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:22:09 (1752510129) [ 435.346261] Key type .llcrypt unregistered [ 435.347451] Key type ._llcrypt unregistered [ 437.081433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 437.157260] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 437.160983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 437.164070] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 437.248312] Key type ._llcrypt registered [ 437.249522] Key type .llcrypt registered [ 437.319632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 437.328578] alg: No test for adler32 (adler32-zlib) [ 438.132579] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 438.135110] LNet: Accept secure, port 988 [ 438.148411] LNet: 51545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.164358] LNet: Removed LNI 192.168.204.24@tcp [ 438.456501] Key type .llcrypt unregistered [ 438.457612] Key type ._llcrypt unregistered [ 440.173214] Key type ._llcrypt registered [ 440.174888] Key type .llcrypt registered [ 440.227986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 440.232687] alg: No test for adler32 (adler32-zlib) [ 441.035577] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 441.037708] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 441.040056] LNet: Accept secure, port 988 [ 441.051914] LNet: 51825:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.053956] LNet: Removed LNI 192.168.204.24@tcp [ 441.060392] LNet: Removed LNI 10.1.2.3@tcp1 [ 441.295309] Key type .llcrypt unregistered [ 441.296381] Key type ._llcrypt unregistered [ 443.160172] Key type ._llcrypt registered [ 443.161597] Key type .llcrypt registered [ 443.247766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.256936] alg: No test for adler32 (adler32-zlib) [ 444.072544] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 444.076159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 444.079605] LNet: Accept secure, port 988 [ 444.101585] LNet: 52094:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.107369] LNet: Removed LNI 192.168.204.24@tcp [ 444.117615] LNet: Removed LNI 10.1.2.3@tcp [ 444.361450] Key type .llcrypt unregistered [ 444.362330] Key type ._llcrypt unregistered [ 446.255405] Key type ._llcrypt registered [ 446.256794] Key type .llcrypt registered [ 446.319413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 446.324171] alg: No test for adler32 (adler32-zlib) [ 447.131811] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 447.133140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 447.134442] LNet: Accept secure, port 988 [ 447.149480] LNet: 52363:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.152407] LNet: Removed LNI 192.168.204.24@tcp [ 447.159469] LNet: Removed LNI 10.1.2.3@tcp [ 447.424415] Key type .llcrypt unregistered [ 447.425440] Key type ._llcrypt unregistered [ 449.563839] Key type ._llcrypt registered [ 449.565424] Key type .llcrypt registered [ 449.622213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.629633] alg: No test for adler32 (adler32-zlib) [ 450.432450] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 450.433779] LNetError: 52639:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 450.435618] LNetError: Error -100 starting up LNI tcp [ 450.442297] LNet: Removed LNI 192.168.204.24@tcp [ 450.682388] Key type .llcrypt unregistered [ 450.683471] Key type ._llcrypt unregistered [ 453.907121] Key type ._llcrypt registered [ 453.908134] Key type .llcrypt registered [ 454.356579] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:22:28 (1752510148) [ 454.615360] Key type .llcrypt unregistered [ 454.616618] Key type ._llcrypt unregistered [ 456.554317] Key type ._llcrypt registered [ 456.555702] Key type .llcrypt registered [ 456.611571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 456.617681] alg: No test for adler32 (adler32-zlib) [ 457.420624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 457.429539] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 457.430939] LNet: Accept secure, port 988 [ 460.134897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 460.338116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e network_timeout [ 460.353118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e network_timeout [ 460.368715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 460.375826] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 460.471963] LNet: 53864:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 460.499526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.512485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 460.597608] LNet: 53898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.605289] LNet: Removed LNI 192.168.204.24@tcp [ 460.752431] Key type .llcrypt unregistered [ 460.753532] Key type ._llcrypt unregistered [ 465.860416] Key type ._llcrypt registered [ 465.861663] Key type .llcrypt registered [ 465.928628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 465.933931] alg: No test for adler32 (adler32-zlib) [ 466.729937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 466.740921] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 466.742428] LNet: Accept secure, port 988 [ 469.329808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 471.705710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.721480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 471.733553] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 472.153239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp -m GET -r 1 -e network_timeout [ 472.169895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.124@tcp1 -m GET -r 1 -e network_timeout [ 472.185767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp -m GET -r 1 -e network_timeout [ 472.199946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -m GET -r 1 -e network_timeout [ 472.214191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp -m GET -r 1 -e network_timeout [ 472.231293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.124@tcp1 -m GET -r 1 -e network_timeout [ 472.247881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -m GET -r 1 -e network_timeout [ 472.265894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -m GET -r 1 -e network_timeout [ 472.284364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 472.294571] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 472.404193] LNet: 54745:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 472.438323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.452479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 474.577121] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 474.687314] LNet: 54813:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.690726] LNet: Removed LNI 192.168.204.24@tcp [ 474.701333] LNet: Removed LNI 192.168.204.24@tcp1 [ 474.869630] Key type .llcrypt unregistered [ 474.871520] Key type ._llcrypt unregistered [ 480.375093] Key type ._llcrypt registered [ 480.376081] Key type .llcrypt registered [ 480.823643] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:22:54 (1752510174) [ 480.936689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 480.942158] alg: No test for adler32 (adler32-zlib) [ 481.738436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.757389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 481.764761] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 481.766889] LNet: Accept secure, port 988 [ 481.986241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 481.993342] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 482.253934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 482.275988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 482.290955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 482.304608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 482.318307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 482.331690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 482.343195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.24@tcp [ 482.352762] LNetError: 55919:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.204.24@tcp to recovery [ 482.783252] LNet: 1 local NIs in recovery (showing 1): 192.168.204.24@tcp [ 482.783289] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 484.831729] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 485.503484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 488.927900] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 489.703438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 489.718940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 489.745300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.753624] LNet: 56036:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.757289] LNet: Removed LNI 192.168.204.24@tcp [ 489.765308] LNet: Removed LNI 192.168.204.24@tcp1 [ 489.773741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.795755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.806579] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 489.808618] LNet: Accept secure, port 988 [ 490.063801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 490.072729] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 490.423766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 490.440535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 490.466494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 490.490167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 490.509652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 490.528387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 490.546830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 490.564667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.24@tcp [ 490.574221] LNetError: 56302:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.204.24@tcp to recovery [ 492.895861] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 492.898319] LNet: Skipped 1 previous similar message [ 493.744247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 496.991783] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 501.017158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 501.030630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 501.044137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 503.175017] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:23:17 (1752510197) [ 503.230872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.242320] LNet: 57018:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.245783] LNet: Removed LNI 192.168.204.24@tcp [ 503.255216] LNet: Removed LNI 192.168.204.24@tcp1 [ 503.263377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.287216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 503.296942] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 503.299757] LNet: Accept secure, port 988 [ 503.544729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 503.554204] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 503.854890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 503.880207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 503.892334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 503.906365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 503.920083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 503.934526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 503.948091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.24@tcp --health 0 [ 503.984457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 504.287231] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.24@tcp [ 506.335853] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 506.337844] LNet: Skipped 2 previous similar messages [ 519.410327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 519.456995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 519.468608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 519.481206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.24@tcp --health 500 [ 519.514731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 526.732277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 526.753159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.761227] LNet: 57520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.775104] LNet: Removed LNI 192.168.204.24@tcp [ 527.786258] LNet: Removed LNI 192.168.204.24@tcp1 [ 527.793254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.812788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.822271] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 527.824263] LNet: Accept secure, port 988 [ 528.083979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 528.093270] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 528.400681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 528.418365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 528.444538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 528.458290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 528.472321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 528.486175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 528.501185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 528.515271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.24@tcp --health 0 [ 529.887257] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 529.889687] LNet: Skipped 2 previous similar messages [ 530.646064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 539.999228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 540.018601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 540.038640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.784820] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:23:56 (1752510236) [ 542.930857] LNet: 58570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.967121] LNet: Removed LNI 192.168.204.24@tcp [ 543.974380] LNet: Removed LNI 192.168.204.24@tcp1 [ 544.195769] Key type .llcrypt unregistered [ 544.197258] Key type ._llcrypt unregistered [ 546.034058] Key type ._llcrypt registered [ 546.035640] Key type .llcrypt registered [ 546.098211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 546.104777] alg: No test for adler32 (adler32-zlib) [ 546.903726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 546.912868] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 546.915050] LNet: Accept secure, port 988 [ 550.255587] Lustre: DEBUG MARKER: Initial discovery [ 550.294398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.124@tcp [ 551.153777] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 551.192250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 551.209350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.124@tcp [ 551.222894] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 551.233923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 551.248842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.124@tcp -r 1 -m GET -l 3 [ 551.265229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.124@tcp -r 1 -m GET -e local_error [ 551.282627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.124@tcp -r 1 -m PUT -l 6 [ 551.796105] Lustre: DEBUG MARKER: Force oleg424-client.virtnet to discover 192.168.204.124@tcp (in background) [ 551.817385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.124@tcp [ 552.338532] Lustre: DEBUG MARKER: Force oleg424-server to discover 192.168.204.24@tcp [ 553.209426] Lustre: DEBUG MARKER: Wait for 59318 [ 554.847289] LNet: There was an unexpected network error while writing to 192.168.204.124: rc = -22 [ 555.329306] Lustre: DEBUG MARKER: Finished wait on 59318 [ 555.347701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 555.366160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 555.458840] LNet: 59774:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.517361] LNet: Removed LNI 192.168.204.24@tcp [ 556.661774] Key type .llcrypt unregistered [ 556.663228] Key type ._llcrypt unregistered [ 561.976311] Key type ._llcrypt registered [ 561.977748] Key type .llcrypt registered [ 562.511706] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:24:16 (1752510256) [ 562.690349] Key type .llcrypt unregistered [ 562.691765] Key type ._llcrypt unregistered [ 564.784101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 564.834490] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 564.836347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 564.838340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 564.946806] Key type ._llcrypt registered [ 564.948735] Key type .llcrypt registered [ 565.012787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 565.019691] alg: No test for adler32 (adler32-zlib) [ 565.823917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.845510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 565.854774] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 565.856428] LNet: Accept secure, port 988 [ 566.115838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 566.125338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 568.876636] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:24:22 (1752510262) [ 568.902814] LNet: 61804:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.904763] LNet: Removed LNI 192.168.204.24@tcp [ 568.911306] LNet: Removed LNI 10.1.2.3@tcp [ 569.152385] Key type .llcrypt unregistered [ 569.153231] Key type ._llcrypt unregistered [ 570.969040] Key type ._llcrypt registered [ 570.969997] Key type .llcrypt registered [ 571.035623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 571.041721] alg: No test for adler32 (adler32-zlib) [ 571.841145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.861071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 571.870149] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 571.872595] LNet: Accept secure, port 988 [ 572.114237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 572.122902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 572.445994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.24@tcp [ 573.471141] LNetError: Unexpected error -22 connecting to 192.168.204.24@tcp at host 192.168.204.24:988 [ 574.514289] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:24:28 (1752510268) [ 574.542254] LNet: 62917:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.583169] LNet: Removed LNI 192.168.204.24@tcp [ 575.590495] LNet: Removed LNI 10.1.2.3@tcp [ 575.817569] Key type .llcrypt unregistered [ 575.818774] Key type ._llcrypt unregistered [ 577.780546] Key type ._llcrypt registered [ 577.782074] Key type .llcrypt registered [ 577.856473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.862940] alg: No test for adler32 (adler32-zlib) [ 578.663425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.684843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.694728] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 578.696694] LNet: Accept secure, port 988 [ 578.943225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 578.952520] LNet: Added LNI 192.168.204.24@tcp2 [8/256/0/180] [ 579.263232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.24@tcp1 --nid 192.168.204.24@tcp2 [ 580.888613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.24@tcp1 192.168.204.24@tcp2 [ 580.899291] LNetError: 63748:0:(lib-move.c:2237:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.24@tcp1 to 192.168.204.24@tcp2 - Route table may be misconfigured [ 580.908256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.24@tcp2 192.168.204.24@tcp1 [ 583.178321] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:24:37 (1752510277) [ 583.220826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.232398] LNet: 64316:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.287162] LNet: Removed LNI 192.168.204.24@tcp1 [ 585.317570] LNet: Removed LNI 192.168.204.24@tcp2 [ 585.326280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.349937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.361724] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 585.363551] LNet: Accept secure, port 988 [ 585.630100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 585.638320] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 585.972437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 585.988967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.24@tcp 192.168.204.24@tcp [ 586.003157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.24@tcp1 192.168.204.24@tcp1 [ 586.018449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.24@tcp -d 192.168.204.24@tcp -e network_timeout [ 586.031657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.24@tcp -d 192.168.204.24@tcp1 -e network_timeout [ 586.045516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp -e network_timeout [ 586.060178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.24@tcp1 -d 192.168.204.24@tcp1 -e network_timeout [ 586.074583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 586.085050] LNet: There was an unexpected network error while writing to 192.168.204.24: rc = -22 [ 586.095258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 586.133430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 586.168224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 586.399099] LNet: 1 local NIs in recovery (showing 1): 192.168.204.24@tcp [ 588.490720] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:24:42 (1752510282) [ 588.525401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.534449] LNet: 65171:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.599158] LNet: Removed LNI 192.168.204.24@tcp [ 590.629491] LNet: Removed LNI 192.168.204.24@tcp1 [ 590.636768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.664945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 590.749854] LNet: 65213:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.890592] Key type .llcrypt unregistered [ 590.891651] Key type ._llcrypt unregistered [ 594.320370] Key type ._llcrypt registered [ 594.322070] Key type .llcrypt registered [ 594.781838] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:24:48 (1752510288) [ 594.919361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.925547] alg: No test for adler32 (adler32-zlib) [ 595.727369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.224446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 596.797183] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:24:50 (1752510290) [ 596.819059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.828720] LNet: 66298:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.841386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.877738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.886613] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 596.888238] LNet: Accept secure, port 988 [ 597.123908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 597.131368] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 597.419786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 597.433657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp1 [ 597.446255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp1 [ 599.498622] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:24:53 (1752510293) [ 600.096046] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg424-server [ 600.566239] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:24:54 (1752510294) [ 601.165829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg424-server [ 601.684366] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:24:55 (1752510295) [ 602.251911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg424-server [ 602.821872] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:24:56 (1752510296) [ 603.365353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg424-server [ 603.886225] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:24:57 (1752510297) [ 604.453890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg424-server [ 604.976739] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:24:58 (1752510298) [ 605.564260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg424-server [ 606.075193] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:25:00 (1752510300) [ 606.623468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg424-server [ 607.157116] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:25:01 (1752510301) [ 607.720720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg424-server [ 608.322147] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:25:02 (1752510302) [ 608.361225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.372833] LNet: 69828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.439133] LNet: Removed LNI 192.168.204.24@tcp [ 610.469789] LNet: Removed LNI 192.168.204.24@tcp1 [ 610.477539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.496711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.505660] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 610.507115] LNet: Accept secure, port 988 [ 610.727678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 610.758733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 610.817405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.824606] LNet: 69978:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.877277] LNet: Removed LNI 192.168.204.24@tcp [ 611.883921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.901917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 611.912617] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 611.914071] LNet: Accept secure, port 988 [ 612.140200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 612.198670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.221386] LNet: Removed LNI 192.168.204.24@tcp [ 613.231186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.256654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.267406] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 613.269799] LNet: Accept secure, port 988 [ 613.545851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 613.581072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 613.652521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.661487] LNet: 70264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.664036] LNet: 70264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 614.693352] LNet: Removed LNI 192.168.204.24@tcp [ 614.702377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.726055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 614.736800] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 614.739814] LNet: Accept secure, port 988 [ 615.029983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 615.093287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.165352] LNet: Removed LNI 192.168.204.24@tcp [ 616.174672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.198676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 616.211269] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 616.213073] LNet: Accept secure, port 988 [ 616.484961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 616.522485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 616.593777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.637412] LNet: Removed LNI 192.168.204.24@tcp [ 617.645963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.668939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 617.679298] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 617.680868] LNet: Accept secure, port 988 [ 617.946666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 618.023808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.034481] LNet: 70693:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.036923] LNet: 70693:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 619.109462] LNet: Removed LNI 192.168.204.24@tcp [ 619.119944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.145567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.157930] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 619.163149] LNet: Accept secure, port 988 [ 619.426061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 619.460463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 619.539071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.581413] LNet: Removed LNI 192.168.204.24@tcp [ 620.590808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.615377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 620.625697] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 620.627194] LNet: Accept secure, port 988 [ 620.913031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 620.995302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.053464] LNet: Removed LNI 192.168.204.24@tcp [ 622.062704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.086566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.098111] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 622.100185] LNet: Accept secure, port 988 [ 622.370530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 622.402551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 622.484785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.525376] LNet: Removed LNI 192.168.204.24@tcp [ 623.534686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.559325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 623.571529] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 623.573481] LNet: Accept secure, port 988 [ 623.861447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 623.952250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.997346] LNet: Removed LNI 192.168.204.24@tcp [ 625.005446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.028290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.038363] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 625.040730] LNet: Accept secure, port 988 [ 625.335944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 625.366835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 626.486317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.495502] LNet: 71420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.497919] LNet: 71420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 627.557405] LNet: Removed LNI 192.168.204.24@tcp [ 627.565641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.590733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 627.601365] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 627.602912] LNet: Accept secure, port 988 [ 627.885629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 627.977560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.029330] LNet: Removed LNI 192.168.204.24@tcp [ 629.038395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.065379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.077646] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 629.080376] LNet: Accept secure, port 988 [ 629.371158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 629.399592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.419571] LNet: Removed LNI 192.168.204.24@tcp [ 629.431488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.462055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.473711] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 629.476177] LNet: Accept secure, port 988 [ 629.756592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 632.194081] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:25:26 (1752510326) [ 632.223628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.242525] LNet: Removed LNI 192.168.204.24@tcp [ 632.251226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.276870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 632.286662] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/180] [ 632.289165] LNet: Accept secure, port 988 [ 632.728775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.744338] LNet: Removed LNI 192.168.204.24@tcp231 [ 632.752598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.776499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 632.784522] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/180] [ 632.786386] LNet: Accept secure, port 988 [ 632.787740] LNet: 72542:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.926170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 632.942472] LNet: Removed LNI 192.168.204.24@tcp231 [ 632.948294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 632.959222] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/180] [ 633.356487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.374304] LNet: Removed LNI 192.168.204.24@tcp231 [ 633.382299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.411634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 633.422749] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/0] [ 633.424292] LNet: Accept secure, port 988 [ 633.425552] LNet: 72691:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 633.560987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 633.579414] LNet: Removed LNI 192.168.204.24@tcp231 [ 633.586455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 633.596576] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/0] [ 633.987080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.998300] LNet: Removed LNI 192.168.204.24@tcp231 [ 634.006244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.035753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 634.047769] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/60] [ 634.050325] LNet: Accept secure, port 988 [ 634.190738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 634.207646] LNet: Removed LNI 192.168.204.24@tcp231 [ 634.220542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 634.239657] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/60] [ 634.635696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.649494] LNet: Removed LNI 192.168.204.24@tcp231 [ 634.659192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.686832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 634.695784] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/180] [ 634.697464] LNet: Accept secure, port 988 [ 634.699034] LNet: 72987:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 634.701475] LNet: 72987:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 634.828939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 634.845392] LNet: Removed LNI 192.168.204.24@tcp231 [ 634.852444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 634.863554] LNet: Added LNI 192.168.204.24@tcp231 [8/256/0/180] [ 637.176325] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:25:31 (1752510331) [ 637.209777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.227067] LNet: Removed LNI 192.168.204.24@tcp231 [ 637.238235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.267517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 637.276225] LNet: Added LNI 192.168.204.24@tcp232 [8/256/0/180] [ 637.278150] LNet: Accept secure, port 988 [ 637.496507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 637.507876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 637.519404] LNet: Removed LNI 192.168.204.24@tcp232 [ 637.525887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 637.534883] LNet: Added LNI 192.168.204.24@tcp232 [8/256/0/180] [ 637.748955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 637.761298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 637.780521] LNet: Removed LNI 192.168.204.24@tcp232 [ 637.785622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 637.792672] LNet: Added LNI 192.168.204.24@tcp232 [8/256/0/180] [ 638.009517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 638.023940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 638.038445] LNet: Removed LNI 192.168.204.24@tcp232 [ 638.044304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 638.053689] LNet: Added LNI 192.168.204.24@tcp232 [8/256/0/180] [ 638.301774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 638.315338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 638.335347] LNet: Removed LNI 192.168.204.24@tcp232 [ 638.341127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 640.326330] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:25:34 (1752510334) [ 640.352038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.367602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.392133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.399890] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 640.401285] LNet: Accept secure, port 988 [ 640.627272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 640.650580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 640.668254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 640.692772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 640.708443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 640.723885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 642.712472] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:25:36 (1752510336) [ 642.740914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.752079] LNet: 75483:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.754899] LNet: 75483:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 642.764609] LNet: Removed LNI 192.168.204.24@tcp [ 642.774812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.794283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.802620] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 642.804321] LNet: Accept secure, port 988 [ 645.032926] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:25:39 (1752510339) [ 645.538203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 646.123556] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:25:40 (1752510340) [ 646.294725] LNet: Removed LNI 192.168.204.24@tcp [ 646.504024] Key type .llcrypt unregistered [ 646.505351] Key type ._llcrypt unregistered [ 648.260099] Key type ._llcrypt registered [ 648.261233] Key type .llcrypt registered [ 648.331821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 648.337594] alg: No test for adler32 (adler32-zlib) [ 649.139817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 649.148071] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 649.149579] LNet: Accept secure, port 988 [ 651.889627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 654.325837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.124@tcp [ 654.428586] LNet: 76931:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.434379] LNet: Removed LNI 192.168.204.24@tcp [ 654.589410] Key type .llcrypt unregistered [ 654.590909] Key type ._llcrypt unregistered [ 658.065585] Key type ._llcrypt registered [ 658.066712] Key type .llcrypt registered [ 658.542878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 659.087331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 659.689994] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:25:53 (1752510353) [ 659.808826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.815100] alg: No test for adler32 (adler32-zlib) [ 660.617783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.649489] LNet: 78013:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.884506] Key type .llcrypt unregistered [ 660.885625] Key type ._llcrypt unregistered [ 662.698436] Key type ._llcrypt registered [ 662.699606] Key type .llcrypt registered [ 662.764140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.769473] alg: No test for adler32 (adler32-zlib) [ 663.584071] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 663.585912] LNet: 78264:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 663.590087] LNet: Accept secure, port 988 [ 663.597515] LNetError: 78276:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 663.608575] LNetError: 78276:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.204.28@tcp has gone from up to down [ 663.671587] LNetError: 78295:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 663.773541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.847138] LNetError: 78276:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.204.27@tcp has gone from up to down [ 666.850789] LNetError: 78276:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 666.853852] LNet: 78320:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.877670] LNet: Removed LNI 192.168.204.24@tcp [ 667.888896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.900946] LNet: 78323:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 670.039627] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:26:03 (1752510363) [ 670.637765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg424-server [ 671.164306] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:26:05 (1752510365) [ 671.699350] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 671.786898] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 671.788615] LNet: 79333:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 671.792234] LNet: 79333:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 671.795758] LNet: Accept secure, port 988 [ 671.802658] LNetError: 79348:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 671.810837] LNetError: 79348:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 671.814499] LNetError: 79348:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.204.28@tcp has gone from up to down [ 671.818383] LNetError: 79348:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 673.383131] Key type lgssc registered [ 673.790142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 674.847503] LNetError: 79339:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.204.27@tcp has gone from up to down [ 674.852377] LNetError: 79339:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 681.611367] Key type lgssc unregistered [ 681.730479] LNet: 80368:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.789382] LNet: Removed LNI 192.168.204.24@tcp [ 683.054583] Key type .llcrypt unregistered [ 683.055967] Key type ._llcrypt unregistered [ 687.985310] Key type ._llcrypt registered [ 687.986237] Key type .llcrypt registered [ 688.460661] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:26:22 (1752510382) [ 688.569395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.575775] alg: No test for adler32 (adler32-zlib) [ 689.437458] LNetError: 81207:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 689.561484] Key type .llcrypt unregistered [ 689.562469] Key type ._llcrypt unregistered [ 691.293697] Key type ._llcrypt registered [ 691.295021] Key type .llcrypt registered [ 691.773775] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:26:25 (1752510385) [ 691.988285] Key type .llcrypt unregistered [ 691.989155] Key type ._llcrypt unregistered [ 693.968369] Key type ._llcrypt registered [ 693.969406] Key type .llcrypt registered [ 694.035771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 694.041452] alg: No test for adler32 (adler32-zlib) [ 697.109722] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:26:31 (1752510391) [ 697.138502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.154091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.184504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.193810] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 697.195723] LNet: Accept secure, port 988 [ 697.448048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.458750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 697.476328] LNet: Removed LNI 192.168.204.24@tcp [ 697.554762] LNet: 82784:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.731438] Key type .llcrypt unregistered [ 697.732848] Key type ._llcrypt unregistered [ 701.254550] Key type ._llcrypt registered [ 701.255591] Key type .llcrypt registered [ 701.694669] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:26:35 (1752510395) [ 701.800456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.805324] alg: No test for adler32 (adler32-zlib) [ 702.605246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.627874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.638242] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 702.640816] LNet: Accept secure, port 988 [ 702.878710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 702.891918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.24@tcp [ 702.899655] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 702.901432] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 702.902961] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 702, 0, 0 [ 702.904950] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 702.906389] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 702.908127] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 702.909592] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 702, 0, 0 [ 702.911960] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 702.913352] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 702.915256] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 702.916791] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 702, 0, 752 [ 702.919505] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 702.920494] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 702.921833] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 702.923075] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 702, 0, 752 [ 702.925107] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 704.990683] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:26:38 (1752510398) [ 705.140132] LNet: 84335:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.149313] LNet: Removed LNI 192.168.204.24@tcp [ 706.398382] Key type .llcrypt unregistered [ 706.399569] Key type ._llcrypt unregistered [ 708.075897] Key type ._llcrypt registered [ 708.077396] Key type .llcrypt registered [ 708.145056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 708.150208] alg: No test for adler32 (adler32-zlib) [ 708.951290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 708.962641] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 708.964800] LNet: Accept secure, port 988 [ 711.668306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 714.110795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.125218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 714.135489] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 714.459721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.462244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.488671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 714.497116] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 717.796657] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:26:51 (1752510411) [ 717.835503] LNet: 85791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.879187] LNet: Removed LNI 192.168.204.24@tcp [ 719.903133] LNet: Removed LNI 192.168.204.24@tcp1 [ 719.910500] LNet: Removed LNI 10.1.2.3@tcp99 [ 720.111530] Key type .llcrypt unregistered [ 720.113076] Key type ._llcrypt unregistered [ 722.156232] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.157519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.241473] Key type ._llcrypt registered [ 722.242556] Key type .llcrypt registered [ 722.313949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 722.321274] alg: No test for adler32 (adler32-zlib) [ 723.125718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.145762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.154388] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 723.156137] LNet: Accept secure, port 988 [ 723.374666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.380979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 724.105494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 724.122771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 724.141786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 724.150761] LNetError: 86505:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 724.162901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 724.179019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 724.319782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 724.344339] LNet: 86525:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.407113] LNet: Removed LNI 192.168.204.24@tcp [ 725.414353] LNet: Removed LNI 10.1.2.3@tcp [ 725.634478] Key type .llcrypt unregistered [ 725.635425] Key type ._llcrypt unregistered [ 729.071115] Key type ._llcrypt registered [ 729.071982] Key type .llcrypt registered [ 729.517167] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:27:03 (1752510423) [ 729.639020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 729.644810] alg: No test for adler32 (adler32-zlib) [ 730.446854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.468656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 730.477268] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 730.478905] LNet: Accept secure, port 988 [ 730.722457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.24@tcp [ 730.751669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg424-client@tcp [ 732.805552] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:27:06 (1752510426) [ 732.836815] LNet: 88042:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.861513] LNet: Removed LNI 192.168.204.24@tcp [ 734.079308] Key type .llcrypt unregistered [ 734.080264] Key type ._llcrypt unregistered [ 735.808371] Key type ._llcrypt registered [ 735.809679] Key type .llcrypt registered [ 735.858726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.863058] alg: No test for adler32 (adler32-zlib) [ 736.662495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 736.673216] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 736.675670] LNet: Accept secure, port 988 [ 736.686763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 738.861887] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:27:12 (1752510432) [ 739.392070] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 739.941766] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:27:13 (1752510433) [ 740.424318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 740.956547] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:27:14 (1752510434) [ 740.985056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.993089] LNet: 89457:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.001523] LNet: Removed LNI 192.168.204.24@tcp [ 741.009786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.037773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 741.050675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 741.269288] Key type .llcrypt unregistered [ 741.270219] Key type ._llcrypt unregistered [ 744.566166] Key type ._llcrypt registered [ 744.567082] Key type .llcrypt registered [ 744.973648] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:27:18 (1752510438) [ 745.090085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.095695] alg: No test for adler32 (adler32-zlib) [ 745.900945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.923948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.933623] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 745.935624] LNet: Accept secure, port 988 [ 746.183640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 746.202960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 748.431559] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:27:22 (1752510442) [ 748.464908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.473697] LNet: 91002:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.541500] LNet: Removed LNI 192.168.204.24@tcp [ 749.549864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.572505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 749.584387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 751.578399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 752.129889] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:27:26 (1752510446) [ 752.158099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.166730] LNet: 91720:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.176649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.255504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.265957] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 752.267834] LNet: Accept secure, port 988 [ 752.524409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 752.534579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.855891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.24@tcp [ 752.873237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 752.901798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.24@tcp -m PUT -e local_timeout -r 1 [ 752.924816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 752.947109] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.949057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.029514] LNetError: 91751:0:(lib-move.c:4921:lnet_parse()) 192.168.204.24@tcp, src 192.168.204.24@tcp: Dropping PUT (error -108 looking up sender) [ 754.079386] LNet: 92027:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.103085] LNet: Removed LNI 192.168.204.24@tcp [ 755.110311] LNet: Removed LNI 10.1.2.3@tcp [ 755.253606] Key type .llcrypt unregistered [ 755.254822] Key type ._llcrypt unregistered [ 758.676868] Key type ._llcrypt registered [ 758.678126] Key type .llcrypt registered [ 759.093463] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:27:33 (1752510453) [ 759.217469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.224130] alg: No test for adler32 (adler32-zlib) [ 760.020849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.140351] LNet: 92875:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.347322] Key type .llcrypt unregistered [ 760.348481] Key type ._llcrypt unregistered [ 762.109648] Key type ._llcrypt registered [ 762.110869] Key type .llcrypt registered [ 762.194842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.199499] alg: No test for adler32 (adler32-zlib) [ 763.003169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 763.013317] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 763.014642] LNet: Accept secure, port 988 [ 765.888934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 765.951670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 766.140544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.124@tcp [ 766.245570] LNet: 93299:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 766.267872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 766.279878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 766.480216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 766.607369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 766.621115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 766.869615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.124@tcp [ 766.978259] LNet: 93485:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 766.980854] LNet: 93485:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 767.011665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 767.026499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 767.253169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 767.376438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 767.387649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 767.508496] LNet: 93619:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.549434] LNet: Removed LNI 192.168.204.24@tcp [ 768.715613] Key type .llcrypt unregistered [ 768.716706] Key type ._llcrypt unregistered [ 773.930550] Key type ._llcrypt registered [ 773.931515] Key type .llcrypt registered [ 774.355087] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:27:48 (1752510468) [ 774.451386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.456634] alg: No test for adler32 (adler32-zlib) [ 775.270948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.419656] LNet: 94503:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.650539] Key type .llcrypt unregistered [ 775.651656] Key type ._llcrypt unregistered [ 777.569175] Key type ._llcrypt registered [ 777.570604] Key type .llcrypt registered [ 777.648288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.656670] alg: No test for adler32 (adler32-zlib) [ 778.463623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 778.475706] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 778.477956] LNet: Accept secure, port 988 [ 781.278548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.124@tcp [ 783.904066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.916790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 783.924514] LNet: Added LNI 192.168.204.24@tcp1 [8/256/0/180] [ 784.211168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 784.371095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.124@tcp [ 784.471283] LNet: 95126:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 784.493314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 784.503777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 784.716171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 784.833281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 784.846992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 785.075680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.124@tcp [ 785.177374] LNet: 95312:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 785.179546] LNet: 95312:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 785.203114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 785.214280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 785.415626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 785.540616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 785.554476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 787.628230] LNet: 95480:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.639156] LNet: Removed LNI 192.168.204.24@tcp [ 788.646321] LNet: Removed LNI 192.168.204.24@tcp1 [ 788.776527] Key type .llcrypt unregistered [ 788.777411] Key type ._llcrypt unregistered [ 794.184650] Key type ._llcrypt registered [ 794.188105] Key type .llcrypt registered [ 794.725685] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 645 sec =========== 12:28:08 (1752510488) [ 794.952363] Key type .llcrypt unregistered [ 794.953406] Key type ._llcrypt unregistered [ 799.078459] Key type ._llcrypt registered [ 799.079457] Key type .llcrypt registered [ 799.279416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.284719] alg: No test for adler32 (adler32-zlib) [ 800.145168] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 800.230316] LNet: Added LNI 192.168.204.24@tcp [8/256/0/180] [ 800.232751] LNet: Accept secure, port 988 [ 801.823203] Key type lgssc registered [ 802.206142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.242165] Lustre: Mounted lustre-client [ 827.712505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 835.073503] LustreError: 98666:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9701c5eec800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 835.080362] LustreError: 98666:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 835.093846] Lustre: Unmounted lustre-client [ 847.266885] Key type lgssc unregistered [ 847.387344] LNet: 99143:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.421766] LNet: Removed LNI 192.168.204.24@tcp [ 848.686845] Key type .llcrypt unregistered [ 848.688321] Key type ._llcrypt unregistered