[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451276859 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003362] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008438] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.011148] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013053] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022865] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027229] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030085] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032103] *** VALIDATE tmpfs *** [ 0.034475] *** VALIDATE proc *** [ 0.035258] *** VALIDATE cgroup *** [ 0.037014] *** VALIDATE cgroup2 *** [ 0.038270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041036] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045110] debug: unmapping init [mem 0xffffffffbb859000-0xffffffffbb860fff] [ 0.047166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048637] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056341] rcu: Hierarchical SRCU implementation. [ 0.058615] smp: Bringing up secondary CPUs ... [ 0.059602] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.064028] smp: Brought up 1 node, 4 CPUs [ 0.066016] smpboot: Max logical packages: 1 [ 0.067020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.278438] node 0 deferred pages initialised in 207ms [ 0.281261] devtmpfs: initialized [ 0.282225] x86/mm: Memory block size: 128MB [ 0.284789] gcov: version magic: 0x41383552 [ 0.286254] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.290118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.293411] pinctrl core: initialized pinctrl subsystem [ 0.295263] [ 0.296010] ************************************************************* [ 0.298018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.300016] ** ** [ 0.301014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.303016] ** ** [ 0.306014] ** This means that this kernel is built to expose internal ** [ 0.308014] ** IOMMU data structures, which may compromise security on ** [ 0.311016] ** your system. ** [ 0.313015] ** ** [ 0.316233] ** If you see this message and you are not debugging the ** [ 0.319018] ** kernel, report this immediately to your vendor! ** [ 0.321015] ** ** [ 0.324018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.326014] ************************************************************* [ 0.329737] NET: Registered protocol family 16 [ 0.331506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.334067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.337071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.341111] cpuidle: using governor menu [ 0.342977] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.343771] PCI: Using configuration type 1 for base access [ 0.345175] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.354116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.356040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.360031] cryptd: max_cpu_qlen set to 1000 [ 0.362238] ACPI: Added _OSI(Module Device) [ 0.363021] ACPI: Added _OSI(Processor Device) [ 0.365022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.366015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.370992] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.377336] ACPI: Interpreter enabled [ 0.378075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.381022] ACPI: Using IOAPIC for interrupt routing [ 0.382142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.386397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.397421] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.401071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.404034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.407133] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.411456] acpiphp: Slot [2] registered [ 0.413325] acpiphp: Slot [5] registered [ 0.415209] acpiphp: Slot [6] registered [ 0.417215] acpiphp: Slot [3] registered [ 0.418174] acpiphp: Slot [4] registered [ 0.420247] acpiphp: Slot [7] registered [ 0.421145] acpiphp: Slot [8] registered [ 0.423180] acpiphp: Slot [9] registered [ 0.425153] acpiphp: Slot [10] registered [ 0.427154] acpiphp: Slot [11] registered [ 0.429208] acpiphp: Slot [12] registered [ 0.431219] acpiphp: Slot [13] registered [ 0.434129] acpiphp: Slot [14] registered [ 0.435120] acpiphp: Slot [15] registered [ 0.436086] acpiphp: Slot [16] registered [ 0.439147] acpiphp: Slot [17] registered [ 0.441140] acpiphp: Slot [18] registered [ 0.442117] acpiphp: Slot [19] registered [ 0.444115] acpiphp: Slot [20] registered [ 0.446115] acpiphp: Slot [21] registered [ 0.447101] acpiphp: Slot [22] registered [ 0.449121] acpiphp: Slot [23] registered [ 0.450145] acpiphp: Slot [24] registered [ 0.452120] acpiphp: Slot [25] registered [ 0.454115] acpiphp: Slot [26] registered [ 0.456196] acpiphp: Slot [27] registered [ 0.457121] acpiphp: Slot [28] registered [ 0.459131] acpiphp: Slot [29] registered [ 0.460095] acpiphp: Slot [30] registered [ 0.462153] acpiphp: Slot [31] registered [ 0.464090] PCI host bridge to bus 0000:00 [ 0.465021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.468023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.470023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.473025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.475024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.478025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.480211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.484025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.488440] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.496589] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.501132] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.506023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.509019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.512025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.516661] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.518770] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.522045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.524826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.529016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.541801] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.545016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.551020] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.562016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.569014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.590014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.598121] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.607028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.615017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.633020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.645568] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648537] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.650341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.660125] iommu: Default domain type: Passthrough [ 0.662514] SCSI subsystem initialized [ 0.664151] ACPI: bus type USB registered [ 0.666110] usbcore: registered new interface driver usbfs [ 0.668202] usbcore: registered new interface driver hub [ 0.670114] usbcore: registered new device driver usb [ 0.673239] pps_core: LinuxPPS API ver. 1 registered [ 0.675010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678067] PTP clock support registered [ 0.681050] EDAC MC: Ver: 3.0.0 [ 0.683176] PCI: Using ACPI for IRQ routing [ 0.684886] NetLabel: Initializing [ 0.685009] NetLabel: domain hash size = 128 [ 0.686010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687116] NetLabel: unlabeled traffic allowed by default [ 0.690043] vgaarb: loaded [ 0.691266] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.692014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.696212] clocksource: Switched to clocksource kvm-clock [ 0.807749] VFS: Disk quotas dquot_6.6.0 [ 0.809514] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.812062] *** VALIDATE ramfs *** [ 0.813282] *** VALIDATE hugetlbfs *** [ 0.814918] pnp: PnP ACPI init [ 0.818255] pnp: PnP ACPI: found 6 devices [ 0.839508] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842496] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.844881] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.847192] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.849701] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.852152] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.855062] NET: Registered protocol family 2 [ 0.857680] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.862690] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.868255] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.873988] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.877470] TCP: Hash tables configured (established 65536 bind 65536) [ 0.880395] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.883535] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885910] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888683] NET: Registered protocol family 1 [ 0.891648] RPC: Registered named UNIX socket transport module. [ 0.893669] RPC: Registered udp transport module. [ 0.895294] RPC: Registered tcp transport module. [ 0.896687] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.899123] NET: Registered protocol family 44 [ 0.900855] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902890] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.904907] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.907430] PCI: CLS 0 bytes, default 64 [ 0.908857] Unpacking initramfs... [ 2.359740] debug: unmapping init [mem 0xffffa0ad7cc64000-0xffffa0ad7ffcffff] [ 2.363813] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.366468] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.369784] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.878980] Initialise system trusted keyrings [ 2.881122] Key type blacklist registered [ 2.883360] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.892602] zbud: loaded [ 2.896883] *** VALIDATE nfs *** [ 2.898478] *** VALIDATE nfs4 *** [ 2.900155] pstore: using deflate compression [ 2.903732] Platform Keyring initialized [ 3.022452] NET: Registered protocol family 38 [ 3.024107] Key type asymmetric registered [ 3.025616] Asymmetric key parser 'x509' registered [ 3.028245] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.031611] io scheduler mq-deadline registered [ 3.033288] io scheduler kyber registered [ 3.035300] io scheduler bfq registered [ 3.037172] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.040328] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.043460] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.046778] ACPI: Power Button [PWRF] [ 3.052767] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.059429] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.075469] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.104312] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.133846] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.140372] Non-volatile memory driver v1.3 [ 3.142782] Linux agpgart interface v0.103 [ 3.178596] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.182134] vda: detected capacity change from 0 to 74698752 [ 3.198242] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.201437] vdb: detected capacity change from 0 to 1073741824 [ 3.212561] libphy: Fixed MDIO Bus: probed [ 3.218406] usbcore: registered new interface driver usbserial_generic [ 3.221568] usbserial: USB Serial support registered for generic [ 3.224353] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.229148] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.231095] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.233813] mousedev: PS/2 mouse device common for all mice [ 3.238283] rtc_cmos 00:05: RTC can wake from S4 [ 3.243188] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.245811] rtc_cmos 00:05: registered as rtc0 [ 3.250695] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.251167] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.257280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.257877] intel_pstate: CPU model not supported [ 3.265542] hid: raw HID events driver (C) Jiri Kosina [ 3.268043] usbcore: registered new interface driver usbhid [ 3.269618] usbhid: USB HID core driver [ 3.271299] drop_monitor: Initializing network drop monitor service [ 3.274192] Initializing XFRM netlink socket [ 3.277372] NET: Registered protocol family 10 [ 3.280854] Segment Routing with IPv6 [ 3.282413] NET: Registered protocol family 17 [ 3.284634] mpls_gso: MPLS GSO support [ 3.289943] RAS: Correctable Errors collector initialized. [ 3.292148] AVX version of gcm_enc/dec engaged. [ 3.294145] AES CTR mode by8 optimization enabled [ 3.373935] sched_clock: Marking stable (3373908350, 0)->(4269021236, -895112886) [ 3.377589] registered taskstats version 1 [ 3.380720] Loading compiled-in X.509 certificates [ 3.382958] zswap: loaded using pool lzo/zbud [ 3.410923] Key type big_key registered [ 3.423911] Key type encrypted registered [ 3.425541] ima: No TPM chip found, activating TPM-bypass! [ 3.427768] ima: Allocated hash algorithm: sha1 [ 3.429751] ima: No architecture policies found [ 3.431700] evm: Initialising EVM extended attributes: [ 3.433784] evm: security.selinux [ 3.435249] evm: security.ima [ 3.437187] evm: security.capability [ 3.438572] evm: HMAC attrs: 0x1 [ 3.441165] rtc_cmos 00:05: setting system clock to 2026-08-12 20:13:18 UTC (1786565598) [ 3.448487] debug: unmapping init [mem 0xffffffffbc803000-0xffffffffbc9fffff] [ 3.452106] debug: unmapping init [mem 0xffffffffbb582000-0xffffffffbb858fff] [ 3.461085] Write protecting the kernel read-only data: 28672k [ 3.464298] debug: unmapping init [mem 0xffffffffb9c03000-0xffffffffb9dfffff] [ 3.467528] debug: unmapping init [mem 0xffffffffba514000-0xffffffffba5fffff] [ 3.499725] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.523844] systemd[1]: Detected virtualization kvm. [ 3.526089] systemd[1]: Detected architecture x86-64. [ 3.527899] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.559374] systemd[1]: No hostname configured. [ 3.561226] systemd[1]: Set hostname to . [ 3.564690] random: systemd: uninitialized urandom read (16 bytes read) [ 3.567137] systemd[1]: Initializing machine ID from random generator. [ 3.630180] random: ln: uninitialized urandom read (6 bytes read) [ 3.724216] random: systemd: uninitialized urandom read (16 bytes read) [ 3.726741] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.731794] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.736676] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.432494] device-mapper: uevent: version 1.0.3 [ 4.435195] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.207524] virtio_net virtio0 ens2: renamed from eth0 [ 5.259912] scsi host0: ata_piix [ 5.414989] scsi host1: ata_piix [ 5.418441] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.420880] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.097101] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.944549] random: crng init done [ 9.946160] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.450390] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.597887] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.846652] SELinux: Disabled at runtime. [ 11.901953] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.911243] systemd[1]: Detected virtualization kvm. [ 11.913187] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.390622] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.394131] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.398988] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.404299] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.407185] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.413225] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.416538] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on[ 12.572349] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.897521] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.212857] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.225513] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.272571] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.293106] EDAC sbridge: Ver: 1.1.2 [ 14.336524] Key type dns_resolver registered [ 14.638522] NFS: Registering the id_resolver key type [ 14.640940] Key type id_resolver registered [ 14.642465] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg455-client login: [ 56.409987] libcfs: loading out-of-tree module taints kernel. [ 56.655610] Key type ._llcrypt registered [ 56.676071] Key type .llcrypt registered [ 57.173372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 57.210839] alg: No test for adler32 (adler32-zlib) [ 58.801825] Lustre: Lustre: Build Version: 2.17.57_1_gc4b410f [ 59.650573] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 61.430356] Key type lgssc registered [ 63.398068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 203.902534] Lustre: Mounted lustre-client - version 2.17.57_1_gc4b410f [ 208.892712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.003328] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.024852] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing yml_node [ 229.344081] Lustre: lustre-OST0000-osc-ffffa0adc5426800: disconnect after 23s idle [ 231.933942] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 234.189959] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 236.802684] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 237.996289] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 12 16:17:11 EDT 2026 [ 254.484591] Lustre: Unmounted lustre-client [ 276.089017] hrtimer: interrupt took 2085968 ns [ 278.979513] Key type lgssc unregistered [ 279.263071] LNet: 8410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.284416] LNetError: 8410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.324797] LNet: Removed LNI 192.168.204.55@tcp [ 280.169161] Key type .llcrypt unregistered [ 280.174191] Key type ._llcrypt unregistered [ 304.067959] Key type ._llcrypt registered [ 304.078111] Key type .llcrypt registered [ 304.343677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 304.357567] alg: No test for adler32 (adler32-zlib) [ 305.283833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 305.366869] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 305.395635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 309.635843] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 309.748671] LNet: 9290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.757728] LNetError: 9290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.773204] LNet: Removed LNI 192.168.204.55@tcp [ 310.429511] Key type .llcrypt unregistered [ 310.432796] Key type ._llcrypt unregistered [ 316.837849] Key type ._llcrypt registered [ 316.846637] Key type .llcrypt registered [ 318.199975] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:18:31 (1786565911) [ 318.510810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.542295] alg: No test for adler32 (adler32-zlib) [ 319.476538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.592780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.617376] LNet: 9713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.637707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.698760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.717791] LNet: 9720:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 327.724499] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:18:41 (1786565921) [ 328.129739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.185797] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.251083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.387688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 336.439738] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:18:50 (1786565930) [ 336.656484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.691151] LNet: 10883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.737827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.865483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 344.007935] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:18:57 (1786565937) [ 344.116790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.152270] LNet: 11464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.193282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.369077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 351.550756] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:19:05 (1786565945) [ 351.621979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.666399] LNet: 12046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.695241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.840893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 352.426717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 359.821646] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:19:13 (1786565953) [ 359.923808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.963161] LNet: 12639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.006586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.118548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.781825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 361.392967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 368.286907] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:19:21 (1786565961) [ 368.385426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.432407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.550252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 375.235901] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:19:29 (1786565969) [ 375.498944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.539853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.591379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 375.630683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.948860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 375.987617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.374145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 376.437385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.777131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 376.816927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.379264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 377.445727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.954475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 377.990079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.474847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 378.518106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.848429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 378.871193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.102435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 379.126886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.652387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 379.699323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.081626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 380.117636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.388527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 380.444647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.806419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 380.847980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.491937] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:19:40 (1786565980) [ 387.678269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.716793] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.729809] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 387.780502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.886469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 387.950613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 394.566682] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:19:48 (1786565988) [ 394.782804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.847648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.963251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.991609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 401.153734] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:19:54 (1786565994) [ 401.257109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.319492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.432794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 401.516623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 407.967844] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:20:01 (1786566001) [ 408.066561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.120925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.225665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 408.252748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 415.442794] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:20:09 (1786566009) [ 415.648585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.669931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.721542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 415.747697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 422.873750] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:20:16 (1786566016) [ 423.088618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.134063] LNet: 17437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.144649] LNet: 17437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 423.208338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.293337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 423.337854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 432.192760] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:20:25 (1786566025) [ 432.320947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.352992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.592634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.045720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.775776] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:20:34 (1786566034) [ 440.924636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.993449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.311799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.675039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.908824] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:20:42 (1786566042) [ 449.066543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.114858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.285661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.509929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.203912] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:20:49 (1786566049) [ 456.311363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.379881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.855475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.087552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.417178] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:20:57 (1786566057) [ 463.741815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.807415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.881297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.922648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.598448] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:21:04 (1786566064) [ 470.846514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.907843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.981526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.060867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.633724] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:21:11 (1786566071) [ 477.784392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.819134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.900810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 484.093757] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:21:18 (1786566078) [ 484.210274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.254992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.357081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 490.085802] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:21:24 (1786566084) [ 490.246101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.263713] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.266908] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 490.284782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.350914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 490.748511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 496.036521] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:21:29 (1786566089) [ 496.125964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.156299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.223408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 496.250846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 502.295573] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:21:36 (1786566096) [ 502.514522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.551054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.609993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 502.651713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 508.263967] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:21:42 (1786566102) [ 508.384277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.422306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.514309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 508.541496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 514.143628] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:21:47 (1786566107) [ 514.225161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.282921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.405200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 514.466310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 520.418980] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:21:54 (1786566114) [ 520.488976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.543256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.620472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 520.694503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 520.721949] LNetError: 25829:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 520.810590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 520.861123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 525.395443] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:21:59 (1786566119) [ 525.601439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.646166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.709267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JrnENjRJlD [ 525.771440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JrnENjRJlD [ 525.815560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JrnENjRJlD [ 530.790201] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:22:04 (1786566124) [ 530.851109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.890107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.972813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 531.041895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 531.093553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 536.166068] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:22:10 (1786566130) [ 536.268302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.311383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.383834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 536.416339] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 537.730729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 538.974681] Lustre: Lustre: Build Version: 2.17.57_1_gc4b410f [ 540.919859] Key type lgssc registered [ 542.078681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 545.586543] LNetError: 27586:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.55@tcp99 has no available nets [ 548.624854] Key type lgssc unregistered [ 548.949577] LNetError: 28700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.966210] LNet: Removed LNI 192.168.204.55@tcp [ 549.545861] Key type .llcrypt unregistered [ 549.551906] Key type ._llcrypt unregistered [ 555.343986] Key type ._llcrypt registered [ 555.354947] Key type .llcrypt registered [ 556.545702] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:22:30 (1786566150) [ 556.876367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 556.888818] alg: No test for adler32 (adler32-zlib) [ 557.809522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.876699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.191451] LNet: Message forwarding will be enabled in 2s [ 563.295131] LNet: Message forwarding enabled [ 563.639940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 563.673712] LNet: Message forwarding will stop in 2s to 6s [ 568.137653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 568.610976] LNet: Message forwarding disabled [ 569.798796] LNet: Message forwarding will be enabled in 2s [ 573.727083] LNet: Message forwarding enabled [ 575.833138] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:22:49 (1786566169) [ 576.230040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.270710] LNet: 30107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.514353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.589228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 583.932610] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:22:57 (1786566177) [ 584.147269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.177913] LNet: 30688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.206617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.263857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 584.299928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 584.346611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 584.374280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 584.410911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 584.460795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 584.481129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 584.509969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 584.544760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 584.572328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 584.601879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 584.634896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 584.663070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 584.702875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 584.730510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 584.756880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 584.787800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 584.819700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 584.852646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 584.875539] LNetError: 30731:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 590.750996] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:23:04 (1786566184) [ 590.935158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.995510] LNet: 31300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.025937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.133517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.367579] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:23:11 (1786566191) [ 598.917715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg455-server [ 600.636575] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:23:14 (1786566194) [ 600.763835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.799975] LNet: 32230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.812828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.863637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.885571] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 601.665740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[56-60]@tcp [ 601.686156] LNet: 32367:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 601.736541] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 601.758033] LNetError: 32237:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.56@tcp has gone from up to down [ 602.734719] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 602.759875] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 603.747198] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 603.760382] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 605.808637] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.57@tcp is being used as a gateway but routing feature is not turned on [ 605.815455] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 607.191895] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:23:21 (1786566201) [ 608.780230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg455-server [ 609.926894] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.58@tcp is being used as a gateway but routing feature is not turned on [ 609.948978] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 610.582282] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:23:24 (1786566204) [ 610.752101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.798584] LNet: 33301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.808035] LNetError: 33301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.831768] LNet: Removed LNI 192.168.204.55@tcp [ 610.860809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.931616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.950799] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 611.872632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[56-61/2]@tcp [ 611.891940] LNet: 33419:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 611.897233] LNet: 33419:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 611.925018] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.56@tcp has gone from up to down [ 611.928909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[56-61/2]@tcp [ 611.935479] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 618.354767] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:23:32 (1786566212) [ 618.437221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.468885] LNetError: 34005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.495886] LNet: Removed LNI 192.168.204.55@tcp [ 618.519308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.616985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 618.676972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.705706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.769445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.852126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.873073] LNet: 34037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.882964] LNet: 34037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 618.919889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.974622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 619.059183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.085502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.156852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.246274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.299537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.354984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 619.431487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.492183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.555954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.634476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.680083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.734573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 619.818763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.862404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.940986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.031848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.060486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.121581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 620.223847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.272500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.380501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.494455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.538508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.585180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 620.646719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.688429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.749281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.241494] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:23:39 (1786566219) [ 627.750971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg455-server [ 629.153534] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:23:43 (1786566223) [ 630.788510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg455-server [ 632.505095] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:23:46 (1786566226) [ 633.392380] Key type .llcrypt unregistered [ 633.394816] Key type ._llcrypt unregistered [ 637.903806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 638.123404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 638.133966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 638.137897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 640.590322] Key type ._llcrypt registered [ 640.591581] Key type .llcrypt registered [ 640.716242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.734928] alg: No test for adler32 (adler32-zlib) [ 641.655922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.719949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.738195] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 642.313271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 642.330822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 643.775355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 643.799075] LNet: Removed LNI 10.1.2.3@tcp [ 643.895875] LNet: 36138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.900838] LNetError: 36138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.908767] LNet: Removed LNI 192.168.204.55@tcp [ 644.414468] Key type .llcrypt unregistered [ 644.417126] Key type ._llcrypt unregistered [ 653.177620] Key type ._llcrypt registered [ 653.180368] Key type .llcrypt registered [ 654.399444] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:24:08 (1786566248) [ 654.891542] Key type .llcrypt unregistered [ 654.894074] Key type ._llcrypt unregistered [ 660.392405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.457031] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.467542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.475107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.097517] Key type ._llcrypt registered [ 663.101340] Key type .llcrypt registered [ 663.320609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.339334] alg: No test for adler32 (adler32-zlib) [ 664.243740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.321674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.364970] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 665.113920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 665.140142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.254462] LNet: 37641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.267474] LNetError: 37641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 667.272669] LNet: Removed LNI 10.1.2.3@tcp [ 667.279397] LNet: Removed LNI 192.168.204.55@tcp [ 667.830691] Key type .llcrypt unregistered [ 667.835146] Key type ._llcrypt unregistered [ 677.400927] Key type ._llcrypt registered [ 677.404496] Key type .llcrypt registered [ 678.503622] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:24:32 (1786566272) [ 678.887788] Key type .llcrypt unregistered [ 678.891370] Key type ._llcrypt unregistered [ 683.285155] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 683.379394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 683.385878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 683.391169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.769781] Key type ._llcrypt registered [ 685.771513] Key type .llcrypt registered [ 685.918501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.937495] alg: No test for adler32 (adler32-zlib) [ 686.800434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.853229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 686.883886] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 688.332616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 688.354280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.860183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.877522] LNet: Removed LNI 10.1.2.3@tcp [ 689.895571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 689.924355] LNet: Removed LNI 10.1.2.31@tcp [ 689.942113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.967696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.544714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 690.561067] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 691.235643] LNet: 39566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.240735] LNetError: 39566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 691.246145] LNet: Removed LNI 10.1.2.31@tcp [ 691.260546] LNet: Removed LNI 10.1.2.3@tcp [ 691.725923] Key type .llcrypt unregistered [ 691.727555] Key type ._llcrypt unregistered [ 700.880624] Key type ._llcrypt registered [ 700.882418] Key type .llcrypt registered [ 701.936532] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:24:55 (1786566295) [ 702.308053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.323255] alg: No test for adler32 (adler32-zlib) [ 703.219404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.258716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.281608] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 703.874740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.892549] LNet: 40521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.897603] LNetError: 40521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.911343] LNet: Removed LNI 192.168.204.55@tcp [ 703.938291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.983588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.55@tcp [ 704.005086] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 704.634454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.55@tcp [ 704.672988] LNet: Removed LNI 192.168.204.55@tcp [ 704.748968] LNet: 40657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.312455] Key type .llcrypt unregistered [ 705.315015] Key type ._llcrypt unregistered [ 714.498464] Key type ._llcrypt registered [ 714.502990] Key type .llcrypt registered [ 715.645555] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:25:09 (1786566309) [ 717.067343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg455-server [ 718.626171] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:25:12 (1786566312) [ 718.994451] Key type .llcrypt unregistered [ 718.996109] Key type ._llcrypt unregistered [ 724.005366] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.081559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.085474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.089554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.648214] Key type ._llcrypt registered [ 726.650265] Key type .llcrypt registered [ 726.805933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.820171] alg: No test for adler32 (adler32-zlib) [ 727.729231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.770132] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 728.445622] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 730.338489] LNet: 42515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.348390] LNetError: 42515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.354280] LNet: Removed LNI 192.168.204.55@tcp1 [ 730.371304] LNet: Removed LNI 10.1.2.3@tcp2 [ 731.023960] Key type .llcrypt unregistered [ 731.028681] Key type ._llcrypt unregistered [ 740.680291] Key type ._llcrypt registered [ 740.683379] Key type .llcrypt registered [ 741.799265] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:25:35 (1786566335) [ 742.272528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 742.297925] alg: No test for adler32 (adler32-zlib) [ 743.202908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.245520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.273840] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 743.940933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.966758] LNet: 43489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.973596] LNetError: 43489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.996866] LNet: Removed LNI 192.168.204.55@tcp [ 744.043386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.087590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 744.103717] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 749.973989] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:25:43 (1786566343) [ 750.158915] LNet: 44080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.165357] LNetError: 44080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.183316] LNet: Removed LNI 192.168.204.55@tcp [ 750.811730] Key type .llcrypt unregistered [ 750.813506] Key type ._llcrypt unregistered [ 755.725606] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.943416] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.949446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.958690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.361290] Key type ._llcrypt registered [ 757.362560] Key type .llcrypt registered [ 757.575962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.588435] alg: No test for adler32 (adler32-zlib) [ 758.534882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.587038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.608370] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 759.254762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.278907] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.226891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.243852] LNet: 44817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.254254] LNetError: 44817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 761.258539] LNet: Removed LNI 10.1.2.3@tcp [ 761.267769] LNet: Removed LNI 192.168.204.55@tcp [ 761.284132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.349354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.367874] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 761.371766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.774282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.808590] LNet: 44844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.815414] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 761.822454] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.828269] LNet: Removed LNI 10.1.2.3@tcp [ 761.837273] LNet: Removed LNI 192.168.204.55@tcp [ 761.858311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.923471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.950396] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 761.956352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.454380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 762.472549] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 763.401754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.419953] LNet: 45007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.424757] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 763.431231] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 763.438926] LNet: Removed LNI 10.1.2.3@tcp [ 763.442486] LNet: Removed LNI 192.168.204.55@tcp [ 763.452323] LNet: Removed LNI 192.168.204.55@tcp2 [ 763.469659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.517946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 763.537834] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 763.543853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.548274] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 769.729269] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:26:03 (1786566363) [ 769.874983] LNet: 45599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.892661] LNetError: 45599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 769.905961] LNetError: 45599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 769.923263] LNet: Removed LNI 10.1.2.3@tcp [ 769.931217] LNet: Removed LNI 192.168.204.55@tcp [ 769.961241] LNet: Removed LNI 192.168.204.55@tcp2 [ 770.550282] Key type .llcrypt unregistered [ 770.555396] Key type ._llcrypt unregistered [ 775.848696] Key type ._llcrypt registered [ 775.852249] Key type .llcrypt registered [ 775.991148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 776.005513] alg: No test for adler32 (adler32-zlib) [ 776.872131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.932839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.970840] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 777.760755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.778195] LNet: 46002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.783868] LNetError: 46002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.795520] LNet: Removed LNI 192.168.204.55@tcp [ 777.813672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.883883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 777.923902] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 778.322435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.339444] LNet: 46032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.344932] LNetError: 46032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.361241] LNet: Removed LNI 192.168.204.55@tcp [ 778.378631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.447274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.468529] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 778.851303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.874853] LNet: Removed LNI 192.168.204.55@tcp [ 778.895896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.951838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.974858] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 779.352224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.471630] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.481226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.489782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.777617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.812464] LNet: 46140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.822609] LNet: 46140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 781.833026] LNetError: 46140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.838793] LNetError: 46140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 781.852606] LNet: Removed LNI 192.168.204.55@tcp [ 781.876557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.944402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.964901] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 788.074382] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:26:21 (1786566381) [ 788.166883] LNet: 46731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.175285] LNetError: 46731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.194953] LNet: Removed LNI 192.168.204.55@tcp [ 788.842933] Key type .llcrypt unregistered [ 788.852232] Key type ._llcrypt unregistered [ 794.863268] Key type ._llcrypt registered [ 794.868116] Key type .llcrypt registered [ 795.047649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.070213] alg: No test for adler32 (adler32-zlib) [ 795.956820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.018517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.066566] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 797.037799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.062596] LNet: 47140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.075255] LNetError: 47140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.097764] LNet: Removed LNI 192.168.204.55@tcp [ 797.123896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.184591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 797.223120] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 802.695431] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:26:36 (1786566396) [ 802.901367] LNet: 47731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.905618] LNetError: 47731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.922717] LNet: Removed LNI 192.168.204.55@tcp [ 803.504195] Key type .llcrypt unregistered [ 803.510329] Key type ._llcrypt unregistered [ 808.735045] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 808.880794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.887256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 808.894420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 811.369537] Key type ._llcrypt registered [ 811.372796] Key type .llcrypt registered [ 811.528461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 811.539391] alg: No test for adler32 (adler32-zlib) [ 812.437342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.488841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.512132] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 813.266970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.283702] LNet: 48184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.287554] LNetError: 48184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.300671] LNet: Removed LNI 192.168.204.55@tcp [ 813.322800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.372521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 813.397239] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 819.134518] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:26:52 (1786566412) [ 819.218298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.249325] LNet: 48776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.256625] LNetError: 48776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.277537] LNet: Removed LNI 192.168.204.55@tcp [ 819.317845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.422557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 824.192878] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:26:58 (1786566418) [ 824.316601] LNet: 49358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.842607] Key type .llcrypt unregistered [ 824.847225] Key type ._llcrypt unregistered [ 829.802841] Key type ._llcrypt registered [ 829.805829] Key type .llcrypt registered [ 830.016214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.027399] alg: No test for adler32 (adler32-zlib) [ 830.939748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.978379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.006257] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 831.805484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.836821] LNet: 49765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.848685] LNetError: 49765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.875875] LNet: Removed LNI 192.168.204.55@tcp [ 831.894907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.967466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 831.994917] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 837.838946] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:27:11 (1786566431) [ 837.943755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.966048] LNet: 50354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.972137] LNetError: 50354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.991434] LNet: Removed LNI 192.168.204.55@tcp [ 838.017664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.083895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 843.732016] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:27:17 (1786566437) [ 845.197655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 846.670444] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:27:20 (1786566440) [ 848.263505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 849.616482] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:27:23 (1786566443) [ 851.027589] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 852.568633] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:27:26 (1786566446) [ 853.759730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 855.177532] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:27:29 (1786566449) [ 856.424590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 857.863775] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:27:31 (1786566451) [ 858.038298] LNet: 52407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.792756] Key type .llcrypt unregistered [ 858.799530] Key type ._llcrypt unregistered [ 864.171706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.258188] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.263885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.271878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.771043] Key type ._llcrypt registered [ 866.773152] Key type .llcrypt registered [ 866.924543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.946499] alg: No test for adler32 (adler32-zlib) [ 867.897094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.034166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.066653] LNet: Added LNI 192.168.204.55@tcp [8/512/0/180] [ 868.072851] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 868.075149] LNet: 52735:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 868.170718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.200124] LNet: 52754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.211708] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.216920] LNet: Removed LNI 10.1.2.3@tcp [ 868.219249] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.237571] LNet: Removed LNI 192.168.204.55@tcp [ 868.266948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.319932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.358073] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 868.417275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.432673] LNet: 52778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.439692] LNetError: 52778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.455736] LNet: Removed LNI 192.168.204.55@tcp [ 868.483428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.586693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.615537] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 868.627759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.631701] LNet: 52786:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 868.740681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.755588] LNet: 52804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.762401] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.769670] LNet: Removed LNI 10.1.2.3@tcp [ 868.771989] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.797035] LNet: Removed LNI 192.168.204.55@tcp [ 868.817284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.858142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.876240] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 868.879223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.880975] LNet: 52812:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 869.032951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.063724] LNet: 52838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.071780] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 869.076391] LNet: Removed LNI 10.1.2.3@tcp [ 869.078363] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.107199] LNet: Removed LNI 192.168.204.55@tcp [ 869.144788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.243229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 869.272761] LNet: Added LNI 192.168.204.55@tcp [8/256/0/182] [ 869.276228] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 869.377863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.399730] LNet: 52866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.404711] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 869.409912] LNet: Removed LNI 10.1.2.3@tcp [ 869.411675] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.427652] LNet: Removed LNI 192.168.204.55@tcp [ 869.454391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.521271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 869.547322] LNet: Added LNI 192.168.204.55@tcp [8/256/0/182] [ 869.552311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 869.554049] LNet: 52874:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 869.621796] LNet: Removed LNI 10.1.2.3@tcp [ 869.629815] LNet: Removed LNI 192.168.204.55@tcp [ 870.252768] Key type .llcrypt unregistered [ 870.255571] Key type ._llcrypt unregistered [ 879.491398] Key type ._llcrypt registered [ 879.495623] Key type .llcrypt registered [ 880.593377] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:27:54 (1786566474) [ 881.892502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 883.433533] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:27:57 (1786566477) [ 883.769870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.787926] alg: No test for adler32 (adler32-zlib) [ 884.713918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.761142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.787477] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 885.375398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.55 [ 885.394195] LNet: 54142:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 885.403868] LNet: Removed LNI 192.168.204.55@tcp [ 890.668987] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:28:04 (1786566484) [ 890.804515] LNet: 54704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.816678] LNetError: 54704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.842533] LNet: Removed LNI 192.168.204.55@tcp [ 891.576813] Key type .llcrypt unregistered [ 891.581231] Key type ._llcrypt unregistered [ 897.210099] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.478138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.244727] Key type ._llcrypt registered [ 900.247899] Key type .llcrypt registered [ 900.411469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.424854] alg: No test for adler32 (adler32-zlib) [ 901.688454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.351757] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:28:21 (1786566501) [ 907.571454] LNet: 55631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.576967] LNetError: 55631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.591893] LNet: Removed LNI 10.1.2.3@tcp [ 908.250460] Key type .llcrypt unregistered [ 908.257219] Key type ._llcrypt unregistered [ 913.689849] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.862981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.867159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.871755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.499145] Key type ._llcrypt registered [ 916.501559] Key type .llcrypt registered [ 916.638651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.649617] alg: No test for adler32 (adler32-zlib) [ 917.563207] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.335501] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:28:36 (1786566516) [ 923.644978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 924.934391] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:28:38 (1786566518) [ 925.109721] LNet: 56818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.118769] LNetError: 56818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 925.147691] LNet: Removed LNI 10.1.2.3@tcp [ 925.809993] Key type .llcrypt unregistered [ 925.812603] Key type ._llcrypt unregistered [ 930.265034] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.507399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.510788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.519690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.186502] Key type ._llcrypt registered [ 933.190098] Key type .llcrypt registered [ 933.370547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.392938] alg: No test for adler32 (adler32-zlib) [ 934.305644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.368939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.303482] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:28:55 (1786566535) [ 941.658819] LNet: 58064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.666615] LNetError: 58064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.685383] LNet: Removed LNI 10.1.2.3@tcp [ 942.337554] Key type .llcrypt unregistered [ 942.343874] Key type ._llcrypt unregistered [ 947.732267] Key type ._llcrypt registered [ 947.735448] Key type .llcrypt registered [ 947.934606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.956506] alg: No test for adler32 (adler32-zlib) [ 948.826979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 948.848486] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 954.591989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 955.080115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_interrupt -r 1 [ 955.115563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_interrupt -r 1 [ 955.142352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 955.156658] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 955.338611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.366445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 955.389902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.416183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.455791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.481735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.501471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.520510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.533870] LNet: 58506:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 955.548219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.574942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.606339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.627575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.653477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.705762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.748397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 956.156533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_dropped -r 1 [ 956.188356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_dropped -r 1 [ 956.225232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 956.246874] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 956.420878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 956.456223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 956.481972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 956.504577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 956.530679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 956.552639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 956.585529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 956.612790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 956.632372] LNet: 58615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.657397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 956.693854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 956.730864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 956.765562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 956.793627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 956.823164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 956.857775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 957.332989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_aborted -r 1 [ 957.361339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_aborted -r 1 [ 957.411706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 957.438413] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 957.628303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.650645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 957.676280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.712194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.754673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.783302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.823014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.865175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.885780] LNet: 58724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.924900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.971826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.017669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.047809] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 958.052393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.080849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.105310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.130748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 958.542633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_no_route -r 1 [ 958.571371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_no_route -r 1 [ 958.595282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 958.813276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.841552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 958.890573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.933401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.972723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 959.003908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.028274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.050470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 959.077408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.100738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.131943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.157082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.181949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.207389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.231333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 959.659650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_timeout -r 1 [ 959.688909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_timeout -r 1 [ 959.716496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 959.736826] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 959.741548] LNet: Skipped 2 previous similar messages [ 959.950883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 959.975531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 960.020491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 960.056904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 960.089262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 960.143063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 960.182112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 960.230024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.261461] LNet: 58943:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 960.267313] LNet: 58943:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 960.288269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.322249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.347155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.382840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.412665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.440670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.468457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 960.843671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_error -r 1 [ 960.878374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_error -r 1 [ 960.915233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 961.138420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.169162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 961.202749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.257760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.291163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.324421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.349326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.374196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.409968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.445628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.486446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.521730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.565709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 961.595438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 961.629219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 961.857157] LNet: 59097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.866895] LNetError: 59097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.886873] LNet: Removed LNI 192.168.204.55@tcp [ 962.210601] Key type .llcrypt unregistered [ 962.218069] Key type ._llcrypt unregistered [ 975.307634] Key type ._llcrypt registered [ 975.309976] Key type .llcrypt registered [ 976.304857] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:29:30 (1786566570) [ 977.135401] Key type .llcrypt unregistered [ 977.138742] Key type ._llcrypt unregistered [ 982.450675] Key type ._llcrypt registered [ 982.455553] Key type .llcrypt registered [ 982.598960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.608687] alg: No test for adler32 (adler32-zlib) [ 983.533311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 983.580857] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 991.155782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1000.484618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.521335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1000.539514] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1001.622568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_interrupt -r 1 [ 1001.659766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_interrupt -r 1 [ 1001.697923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_interrupt -r 1 [ 1001.749154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_interrupt -r 1 [ 1001.788470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_interrupt -r 1 [ 1001.826319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_interrupt -r 1 [ 1001.865636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_interrupt -r 1 [ 1001.910383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_interrupt -r 1 [ 1001.948979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1001.971547] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1001.977787] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1002.175885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.204196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.239537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.274878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.301838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.356996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.394548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.432753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.456334] LNet: 60568:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1002.478736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.505752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.534511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.565844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.588476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.620784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.655495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1003.090980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_dropped -r 1 [ 1003.128734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_dropped -r 1 [ 1003.157191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_dropped -r 1 [ 1003.185971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_dropped -r 1 [ 1003.217878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_dropped -r 1 [ 1003.249930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_dropped -r 1 [ 1003.287775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_dropped -r 1 [ 1003.321291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_dropped -r 1 [ 1003.356560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1003.371690] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1003.375923] LNet: Skipped 2 previous similar messages [ 1003.520644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.544766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1003.576385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.611434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1003.635535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.656662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.683303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.716808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.730338] LNet: 60689:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.747945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.768946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.793186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.817709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.839821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.861273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.883655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1004.242487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_aborted -r 1 [ 1004.266771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_aborted -r 1 [ 1004.295603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_aborted -r 1 [ 1004.341774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_aborted -r 1 [ 1004.388846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_aborted -r 1 [ 1004.423301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_aborted -r 1 [ 1004.452606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_aborted -r 1 [ 1004.497969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_aborted -r 1 [ 1004.529996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1004.551996] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1004.563481] LNet: Skipped 2 previous similar messages [ 1004.722660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.755557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1004.784924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1004.811332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.832774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.862571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.884674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1004.909651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1004.922265] LNet: 60810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1004.937386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1004.968776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1004.990526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.012419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.037637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.062043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.091781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1005.433067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_no_route -r 1 [ 1005.458685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_no_route -r 1 [ 1005.485453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_no_route -r 1 [ 1005.511391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_no_route -r 1 [ 1005.542849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_no_route -r 1 [ 1005.571674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_no_route -r 1 [ 1005.604885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_no_route -r 1 [ 1005.629837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_no_route -r 1 [ 1005.664441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1005.819418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1005.849477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1005.882448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.911184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.941922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1005.962593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.989397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.011941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.040395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.065257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.104477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.130968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.159683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.194939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.223777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1006.580671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_timeout -r 1 [ 1006.605819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_timeout -r 1 [ 1006.631645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_timeout -r 1 [ 1006.658294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_timeout -r 1 [ 1006.683801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_timeout -r 1 [ 1006.716172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_timeout -r 1 [ 1006.743614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_timeout -r 1 [ 1006.771304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_timeout -r 1 [ 1006.801765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1006.817230] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1006.820929] LNet: Skipped 5 previous similar messages [ 1006.970099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.995657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1007.018494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.044066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.067911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.083915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1007.107715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1007.135539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1007.151812] LNet: 61053:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1007.157454] LNet: 61053:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1007.178421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.204606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.234367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.253709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.277925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.303483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.327472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.743900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_error -r 1 [ 1007.778263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_error -r 1 [ 1007.814709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_error -r 1 [ 1007.848647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_error -r 1 [ 1007.902567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_error -r 1 [ 1007.934678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_error -r 1 [ 1007.973908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_error -r 1 [ 1008.003578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_error -r 1 [ 1008.024972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1008.152165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1008.177656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1008.198201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.217746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.239265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1008.259662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.279485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.307625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.335571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.360347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.401640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.441648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.483635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.513758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.555447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1012.392967] LNet: 61254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.400861] LNetError: 61254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.410867] LNet: Removed LNI 192.168.204.55@tcp [ 1012.423967] LNet: Removed LNI 192.168.204.55@tcp1 [ 1012.780783] Key type .llcrypt unregistered [ 1012.785470] Key type ._llcrypt unregistered [ 1024.977896] Key type ._llcrypt registered [ 1024.979609] Key type .llcrypt registered [ 1025.864384] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:30:19 (1786566619) [ 1026.511455] Key type .llcrypt unregistered [ 1026.513185] Key type ._llcrypt unregistered [ 1030.716319] Key type ._llcrypt registered [ 1030.718156] Key type .llcrypt registered [ 1030.860881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.873416] alg: No test for adler32 (adler32-zlib) [ 1031.751480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1031.771640] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1037.804305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1038.405975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_dropped -r 1 [ 1038.459443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_dropped -r 1 [ 1038.521145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1038.546986] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1038.777922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1038.804339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1038.835712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.868465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.902316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.940414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.977973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.009888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.031276] LNet: 62507:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1039.055416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.079980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.100373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.124703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.153976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.193298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.231325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1039.758941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_error -r 1 [ 1039.794195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_error -r 1 [ 1039.835636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1039.863501] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1040.044459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.102195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1040.135440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.172106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.208784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.247141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.289379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.345098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.369645] LNet: 62616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.407861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.436846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.460432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.498236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.529325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.565365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.615283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1041.126960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_timeout -r 1 [ 1041.166995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_timeout -r 1 [ 1041.213822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1041.233338] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1041.457383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.498616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.532227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.577934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.615129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.646348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.683242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.739991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.754478] LNet: 62725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.771758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.798661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.831528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.864681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.897425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.931843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.966577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1042.200581] LNet: 62771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.209470] LNetError: 62771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.228045] LNet: Removed LNI 192.168.204.55@tcp [ 1042.535377] Key type .llcrypt unregistered [ 1042.540747] Key type ._llcrypt unregistered [ 1053.705543] Key type ._llcrypt registered [ 1053.707917] Key type .llcrypt registered [ 1054.667465] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:30:48 (1786566648) [ 1055.324860] Key type .llcrypt unregistered [ 1055.327164] Key type ._llcrypt unregistered [ 1058.660804] Key type ._llcrypt registered [ 1058.664462] Key type .llcrypt registered [ 1058.785818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.796268] alg: No test for adler32 (adler32-zlib) [ 1059.668724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1059.693735] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1064.589766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1069.843287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.880675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1069.903910] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1070.856611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_dropped -r 1 [ 1070.884703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e remote_dropped -r 1 [ 1070.908935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_dropped -r 1 [ 1070.942809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e remote_dropped -r 1 [ 1070.976074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e remote_dropped -r 1 [ 1070.997861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e remote_dropped -r 1 [ 1071.024790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e remote_dropped -r 1 [ 1071.061565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e remote_dropped -r 1 [ 1071.089633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1071.113488] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1071.118630] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.155@tcp [ 1071.264759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.290239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.314325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.336781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.367928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.394719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.419539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.443398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.459291] LNet: 64240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1071.479736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.518409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.544378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.579719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.610172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.643379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.669652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1071.695583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.014657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_error -r 1 [ 1072.044606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e remote_error -r 1 [ 1072.071516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_error -r 1 [ 1072.103614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e remote_error -r 1 [ 1072.148276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e remote_error -r 1 [ 1072.191505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e remote_error -r 1 [ 1072.228549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e remote_error -r 1 [ 1072.260972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e remote_error -r 1 [ 1072.291750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1072.308924] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1072.315095] LNet: Skipped 2 previous similar messages [ 1072.473047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.503521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.536115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.554448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.583615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.607232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.632545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.650872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.664932] LNet: 64364:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.679323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.703265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.726464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.756595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.789144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.813562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.842951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1072.868945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.191801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_timeout -r 1 [ 1073.224712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e remote_timeout -r 1 [ 1073.258046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_timeout -r 1 [ 1073.289900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e remote_timeout -r 1 [ 1073.326153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e remote_timeout -r 1 [ 1073.369396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e remote_timeout -r 1 [ 1073.399615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e remote_timeout -r 1 [ 1073.429312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e remote_timeout -r 1 [ 1073.461834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.155@tcp [ 1073.478433] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1073.603938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.634308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.666534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.693471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.719870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.748877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.776539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.813173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.834672] LNet: 64487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1073.855589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.888785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.919358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.945606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.972335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.994519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.016087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1074.035799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.747675] LNet: 64569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.753290] LNetError: 64569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.758316] LNet: Removed LNI 192.168.204.55@tcp [ 1077.768513] LNet: Removed LNI 192.168.204.55@tcp1 [ 1078.161620] Key type .llcrypt unregistered [ 1078.163420] Key type ._llcrypt unregistered [ 1090.799770] Key type ._llcrypt registered [ 1090.803864] Key type .llcrypt registered [ 1092.010382] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:31:25 (1786566685) [ 1093.329445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1094.804556] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:31:28 (1786566688) [ 1095.549057] Key type .llcrypt unregistered [ 1095.554576] Key type ._llcrypt unregistered [ 1100.643872] Key type ._llcrypt registered [ 1100.646192] Key type .llcrypt registered [ 1100.830150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.853418] alg: No test for adler32 (adler32-zlib) [ 1101.729079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.753027] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1108.160902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1108.874970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e network_timeout -r 1 [ 1108.914433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e network_timeout -r 1 [ 1108.952508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1108.994080] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1109.227046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.261207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.291218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.327951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.366810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.406933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.439986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.469793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.485340] LNet: 66116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.499261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.529376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.562395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.588881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.620561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.670773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.970726] LNet: 66159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.979987] LNetError: 66159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.992963] LNet: Removed LNI 192.168.204.55@tcp [ 1110.329174] Key type .llcrypt unregistered [ 1110.333355] Key type ._llcrypt unregistered [ 1124.749135] Key type ._llcrypt registered [ 1124.755740] Key type .llcrypt registered [ 1124.984139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.004770] alg: No test for adler32 (adler32-zlib) [ 1125.902800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1125.937288] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1132.267152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1139.183832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.219994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1139.240804] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1140.359726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e network_timeout -r 1 [ 1140.403193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e network_timeout -r 1 [ 1140.432714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e network_timeout -r 1 [ 1140.455132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e network_timeout -r 1 [ 1140.485795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e network_timeout -r 1 [ 1140.511345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e network_timeout -r 1 [ 1140.539913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e network_timeout -r 1 [ 1140.575598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e network_timeout -r 1 [ 1140.625314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1140.643582] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1140.797835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.824067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.852101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.874847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.899457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.944311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.964115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.987227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.001482] LNet: 67042:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.021722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.048683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.078027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.102276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.123303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.152571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.781197] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1145.097624] LNet: 67120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.108676] LNetError: 67120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.118286] LNet: Removed LNI 192.168.204.55@tcp [ 1145.142192] LNet: Removed LNI 192.168.204.55@tcp1 [ 1145.536635] Key type .llcrypt unregistered [ 1145.539894] Key type ._llcrypt unregistered [ 1158.613910] Key type ._llcrypt registered [ 1158.616498] Key type .llcrypt registered [ 1159.863542] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:32:33 (1786566753) [ 1160.318487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1160.336875] alg: No test for adler32 (adler32-zlib) [ 1161.267366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.309484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1161.326982] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1161.957564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1161.972388] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1162.664429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1162.712596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1162.747259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1162.795928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1162.825293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1162.874223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1162.920442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.55@tcp [ 1162.947973] LNetError: 68219:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.55@tcp to recovery [ 1163.359708] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1163.359812] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1165.407397] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1166.369816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1169.503460] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1169.785781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1169.815859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.863468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1169.881888] LNet: 68326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.888827] LNetError: 68326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.911549] LNet: Removed LNI 192.168.204.55@tcp [ 1170.919427] LNetError: 68326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.942869] LNet: Removed LNI 192.168.204.55@tcp1 [ 1170.965263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.012830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1171.031460] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1171.771618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1171.797872] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1172.569610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1172.602590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.659569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1172.724335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1172.776494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1172.828369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1172.876323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1172.916576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.55@tcp [ 1172.952969] LNetError: 68599:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.55@tcp to recovery [ 1173.023721] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1175.457514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1179.167451] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1179.172504] LNet: Skipped 1 previous similar message [ 1184.431767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1184.480129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.505868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.316989] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:33:04 (1786566784) [ 1190.411328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1190.426785] LNet: 69311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.441472] LNetError: 69311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.448304] LNet: Removed LNI 192.168.204.55@tcp [ 1190.479198] LNet: Removed LNI 192.168.204.55@tcp1 [ 1190.503983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.546724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1190.562388] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1191.143454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1191.164365] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1191.980560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1192.072883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1192.098257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1192.126545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1192.149778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1192.181832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1192.215925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.55@tcp --health 0 [ 1192.283326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1192.607176] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.55@tcp [ 1193.631466] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1193.635562] LNet: Skipped 1 previous similar message [ 1208.581864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1208.712897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.747361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.778527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.55@tcp --health 500 [ 1208.850272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1215.355639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1215.400324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1215.414210] LNet: 69816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.418690] LNetError: 69816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.422931] LNetError: 69816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1215.426978] LNet: Removed LNI 192.168.204.55@tcp [ 1215.440269] LNet: Removed LNI 192.168.204.55@tcp1 [ 1215.451692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.499693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1215.534271] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1216.098802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1216.117703] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1216.760254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1216.795386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.842501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1216.866813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1216.890589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1216.926761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1216.957633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1216.998794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.55@tcp --health 0 [ 1218.527449] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1218.538945] LNet: Skipped 3 previous similar messages [ 1219.357044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1229.408890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1229.431502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.456551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.779989] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:33:48 (1786566828) [ 1235.072411] LNet: 70873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.076979] LNetError: 70873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.085996] LNetError: 70873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1235.093908] LNet: Removed LNI 192.168.204.55@tcp [ 1235.106036] LNet: Removed LNI 192.168.204.55@tcp1 [ 1235.657168] Key type .llcrypt unregistered [ 1235.659729] Key type ._llcrypt unregistered [ 1240.479363] Key type ._llcrypt registered [ 1240.481555] Key type .llcrypt registered [ 1240.668168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.695897] alg: No test for adler32 (adler32-zlib) [ 1241.586315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1241.619498] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1248.426975] Lustre: DEBUG MARKER: Initial discovery [ 1248.541102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.155@tcp [ 1250.286934] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1250.366308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1250.401213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.155@tcp [ 1250.422737] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1250.440179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.468363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.155@tcp -r 1 -m GET -l 3 [ 1250.500689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.155@tcp -r 1 -m GET -e local_error [ 1250.533914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.155@tcp -r 1 -m PUT -l 6 [ 1250.847986] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1251.476361] Lustre: DEBUG MARKER: Force oleg455-client.virtnet to discover 192.168.204.155@tcp (in background) [ 1251.659154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.155@tcp [ 1252.655307] Lustre: DEBUG MARKER: Force oleg455-server to discover 192.168.204.55@tcp [ 1254.336351] Lustre: DEBUG MARKER: Wait for 71640 [ 1254.687422] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1255.649173] Lustre: DEBUG MARKER: Finished wait on 71640 [ 1255.748382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1255.766021] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1255.780970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.977342] LNet: 72092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.981994] LNetError: 72092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.000451] LNet: Removed LNI 192.168.204.55@tcp [ 1256.263681] Key type .llcrypt unregistered [ 1256.267523] Key type ._llcrypt unregistered [ 1266.246316] Key type ._llcrypt registered [ 1266.247919] Key type .llcrypt registered [ 1267.131910] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:34:21 (1786566861) [ 1267.572486] Key type .llcrypt unregistered [ 1267.575113] Key type ._llcrypt unregistered [ 1271.602218] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1271.690245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1271.694068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1271.699837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1274.003688] Key type ._llcrypt registered [ 1274.005658] Key type .llcrypt registered [ 1274.123241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.130843] alg: No test for adler32 (adler32-zlib) [ 1274.985663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.022701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.040865] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1275.520238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1275.538520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1280.771586] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:34:34 (1786566874) [ 1280.901114] LNet: 74172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.907915] LNetError: 74172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1280.914697] LNet: Removed LNI 10.1.2.3@tcp [ 1280.926372] LNet: Removed LNI 192.168.204.55@tcp [ 1281.405054] Key type .llcrypt unregistered [ 1281.409731] Key type ._llcrypt unregistered [ 1285.618046] Key type ._llcrypt registered [ 1285.619412] Key type .llcrypt registered [ 1285.742701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.750105] alg: No test for adler32 (adler32-zlib) [ 1286.610802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.670483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1286.686500] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1287.223726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1287.238975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1287.892552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.55@tcp [ 1288.927151] LNetError: Unexpected error -22 connecting to 192.168.204.55@tcp at host 192.168.204.55:988 [ 1292.154718] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:34:46 (1786566886) [ 1292.296747] LNet: 75306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.314664] LNetError: 75306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1292.335888] LNet: Removed LNI 10.1.2.3@tcp [ 1293.350789] LNet: Removed LNI 192.168.204.55@tcp [ 1293.871674] Key type .llcrypt unregistered [ 1293.873908] Key type ._llcrypt unregistered [ 1298.054084] Key type ._llcrypt registered [ 1298.056191] Key type .llcrypt registered [ 1298.189512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1298.204826] alg: No test for adler32 (adler32-zlib) [ 1299.056158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.097183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.118380] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1299.576108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1299.592384] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 1300.188328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.55@tcp1 --nid 192.168.204.55@tcp2 [ 1302.914396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp1 192.168.204.55@tcp2 [ 1302.926157] LNetError: 76146:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.55@tcp1 to 192.168.204.55@tcp2 - Route table may be misconfigured [ 1302.952154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp2 192.168.204.55@tcp1 [ 1306.994463] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:35:01 (1786566901) [ 1307.101798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1307.126679] LNet: 76717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.135460] LNetError: 76717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.191153] LNet: Removed LNI 192.168.204.55@tcp1 [ 1308.193183] LNetError: 76717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.209420] LNet: Removed LNI 192.168.204.55@tcp2 [ 1308.224903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.272897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.293520] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1308.798446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1308.815958] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1309.492444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1309.532447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.55@tcp [ 1309.574162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp1 192.168.204.55@tcp1 [ 1309.609577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.55@tcp -d 192.168.204.55@tcp -e network_timeout [ 1309.644814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -e network_timeout [ 1309.668659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -e network_timeout [ 1309.700141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -e network_timeout [ 1309.726615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1309.742994] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1309.766825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1309.856808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1309.958408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1310.303539] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1314.401438] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:35:08 (1786566908) [ 1314.505720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.521530] LNet: 77576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.529429] LNetError: 77576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.539792] LNet: Removed LNI 192.168.204.55@tcp [ 1315.576304] LNet: Removed LNI 192.168.204.55@tcp1 [ 1315.596572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.659887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1315.868114] LNet: 77622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.124540] Key type .llcrypt unregistered [ 1316.126494] Key type ._llcrypt unregistered [ 1324.029902] Key type ._llcrypt registered [ 1324.031607] Key type .llcrypt registered [ 1325.023147] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:35:18 (1786566918) [ 1325.306477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.319340] alg: No test for adler32 (adler32-zlib) [ 1326.190359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.281598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1328.547499] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:35:22 (1786566922) [ 1328.624406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.644032] LNet: 78714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.664780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.725209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.748207] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1329.272716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.291964] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1330.028660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1330.067377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp1 [ 1330.090931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp1 [ 1334.563434] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:35:28 (1786566928) [ 1335.855128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg455-server [ 1337.022593] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:35:31 (1786566931) [ 1338.323219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg455-server [ 1339.324210] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:35:33 (1786566933) [ 1340.403639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg455-server [ 1341.531399] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:35:35 (1786566935) [ 1342.605245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg455-server [ 1343.717709] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:35:37 (1786566937) [ 1344.907554] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg455-server [ 1345.965463] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:35:40 (1786566940) [ 1347.295574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg455-server [ 1348.433895] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:35:42 (1786566942) [ 1349.880549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg455-server [ 1351.135539] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:35:45 (1786566945) [ 1352.514530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg455-server [ 1353.745916] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:35:47 (1786566947) [ 1355.194050] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg455-server [ 1356.407579] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:35:50 (1786566950) [ 1356.618804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.649274] LNet: 82594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.659274] LNetError: 82594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.727467] LNet: Removed LNI 192.168.204.55@tcp [ 1357.743515] LNetError: 82594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.761032] LNet: Removed LNI 192.168.204.55@tcp1 [ 1357.781353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.865217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.884132] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1358.530245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1358.597515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1358.732629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.752556] LNet: 82742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.759842] LNetError: 82742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.777652] LNet: Removed LNI 192.168.204.55@tcp [ 1358.801536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.858471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1358.885472] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1359.465529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1359.582513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1360.614716] LNet: Removed LNI 192.168.204.55@tcp [ 1360.638352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.704661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.730854] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1361.249770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1361.314929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1362.542738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.559631] LNet: 83044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.562680] LNet: 83044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1362.568108] LNetError: 83044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.573754] LNetError: 83044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1362.590617] LNet: Removed LNI 192.168.204.55@tcp [ 1362.610796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.651929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1362.671075] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1363.192680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1363.313425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.390721] LNet: Removed LNI 192.168.204.55@tcp [ 1364.412351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.460795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.481256] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1365.110419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1365.202664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1365.374679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.438964] LNet: Removed LNI 192.168.204.55@tcp [ 1366.473970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.542171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1366.576527] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1367.195585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1367.430215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.443434] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.446641] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1367.452321] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.464112] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1367.488246] LNet: Removed LNI 192.168.204.55@tcp [ 1367.505915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.534255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1367.548746] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1368.057665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1368.124839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1369.766347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.883345] LNet: Removed LNI 192.168.204.55@tcp [ 1370.906542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.984571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1371.004525] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1371.931515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1372.237561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.350058] LNet: Removed LNI 192.168.204.55@tcp [ 1373.387443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.474742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.509230] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1374.377158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1374.534865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1374.847183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1375.915132] LNet: Removed LNI 192.168.204.55@tcp [ 1375.943857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.041850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1376.109686] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1377.593736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1378.021717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.078148] LNet: 84076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.093447] LNet: 84076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1378.110615] LNetError: 84076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.122268] LNetError: 84076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1379.175459] LNet: Removed LNI 192.168.204.55@tcp [ 1379.239281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.345993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.394332] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1380.742829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1380.895811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1381.388262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.544611] LNet: Removed LNI 192.168.204.55@tcp [ 1382.599907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.694941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1382.769765] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1383.718887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1384.039920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.125978] LNet: Removed LNI 192.168.204.55@tcp [ 1385.164562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.240437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.277475] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1385.904613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1386.007121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.067847] LNet: Removed LNI 192.168.204.55@tcp [ 1386.164189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.342132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.400417] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1387.620231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1387.697932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.721844] LNet: Removed LNI 192.168.204.55@tcp [ 1387.737085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.781249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1387.855543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.908918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.984539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1388.240145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.344171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.522852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1388.735082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.842649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.031683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1389.084717] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1390.468418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.541621] LNet: Removed LNI 192.168.204.55@tcp [ 1390.577177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.725791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1405.276660] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:36:37 (1786566997) [ 1405.512377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1405.576803] LNet: 85386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.596514] LNet: 85386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1405.653592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.840067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1405.899043] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/180] [ 1408.668459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.757594] LNetError: 85530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.768772] LNetError: 85530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1408.801077] LNet: Removed LNI 192.168.204.55@tcp231 [ 1408.884634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.054211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1409.102657] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/180] [ 1409.111889] LNet: 85539:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.029123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1410.103724] LNet: Removed LNI 192.168.204.55@tcp231 [ 1410.141463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1410.232337] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/180] [ 1412.375042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.420278] LNet: Removed LNI 192.168.204.55@tcp231 [ 1412.451801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.646895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1412.701151] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/0] [ 1412.712758] LNet: 85691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1413.556099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1413.611295] LNet: Removed LNI 192.168.204.55@tcp231 [ 1413.653232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1413.691791] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/0] [ 1415.469929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.510777] LNet: Removed LNI 192.168.204.55@tcp231 [ 1415.533115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.597521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1415.635632] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/60] [ 1415.644746] LNet: 85842:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1416.219575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1416.243977] LNet: Removed LNI 192.168.204.55@tcp231 [ 1416.263293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1416.299774] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/60] [ 1417.830739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.864354] LNet: Removed LNI 192.168.204.55@tcp231 [ 1417.913550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.053616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1418.102586] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/180] [ 1418.110791] LNet: 85994:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1419.019423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1419.062408] LNet: Removed LNI 192.168.204.55@tcp231 [ 1419.083861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1419.119929] LNet: Added LNI 192.168.204.55@tcp231 [8/256/0/180] [ 1429.467499] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:37:02 (1786567022) [ 1429.577895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.627198] LNet: Removed LNI 192.168.204.55@tcp231 [ 1429.685401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.814170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1429.863879] LNet: Added LNI 192.168.204.55@tcp232 [8/256/0/180] [ 1431.048577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1431.118148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1431.146698] LNet: Removed LNI 192.168.204.55@tcp232 [ 1431.161713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1431.183189] LNet: Added LNI 192.168.204.55@tcp232 [8/256/0/180] [ 1432.286797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1432.347761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1432.402955] LNet: Removed LNI 192.168.204.55@tcp232 [ 1432.437758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1432.494714] LNet: Added LNI 192.168.204.55@tcp232 [8/256/0/180] [ 1433.611087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1433.647119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1433.719218] LNet: Removed LNI 192.168.204.55@tcp232 [ 1433.752131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1433.803411] LNet: Added LNI 192.168.204.55@tcp232 [8/256/0/180] [ 1434.861626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1434.934529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1434.970649] LNet: Removed LNI 192.168.204.55@tcp232 [ 1434.986947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1445.131632] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:37:18 (1786567038) [ 1445.955575] LNet: 87834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.963771] LNet: 87834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1447.069137] Key type .llcrypt unregistered [ 1447.071047] Key type ._llcrypt unregistered [ 1459.772529] Key type ._llcrypt registered [ 1459.779855] Key type .llcrypt registered [ 1459.946514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.982212] alg: No test for adler32 (adler32-zlib) [ 1460.948072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.966333] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1470.595721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1482.654454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.712061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.738897] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1484.290761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_interrupt -r 2 [ 1484.326751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_interrupt -r 2 [ 1484.385319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_interrupt -r 2 [ 1484.437774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_interrupt -r 2 [ 1484.501086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_interrupt -r 2 [ 1484.546654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_interrupt -r 2 [ 1484.606607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_interrupt -r 2 [ 1484.646950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_interrupt -r 2 [ 1484.695315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1484.728910] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1484.742280] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1485.126864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.187139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1485.262750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.333241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.390685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.432794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.474548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.504223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.547664] LNet: 88502:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1485.572951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.613591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.657652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.704511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.754705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.813201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.499520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_dropped -r 2 [ 1486.559105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_dropped -r 2 [ 1486.607964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_dropped -r 2 [ 1486.651756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_dropped -r 2 [ 1486.711835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_dropped -r 2 [ 1486.775564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_dropped -r 2 [ 1486.823742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_dropped -r 2 [ 1486.889075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_dropped -r 2 [ 1486.977314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1487.018215] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1487.308944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.336454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1487.365912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.395687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.419486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1487.443820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1487.485476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1487.531343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1487.558170] LNet: 88622:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1487.580133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1487.614423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1487.659471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1487.706878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1487.751220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1487.789563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.485182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_aborted -r 2 [ 1488.524328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_aborted -r 2 [ 1488.559709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_aborted -r 2 [ 1488.610443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_aborted -r 2 [ 1488.676700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_aborted -r 2 [ 1488.749299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_aborted -r 2 [ 1488.812514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_aborted -r 2 [ 1488.865560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_aborted -r 2 [ 1488.911914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1488.937933] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1489.280221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1489.315889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1489.352620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1489.396913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1489.430819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1489.455809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1489.484672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1489.510901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1489.525688] LNet: 88743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1489.542411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1489.569746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1489.595798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.628853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1489.655897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1489.681582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1490.210839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_no_route -r 2 [ 1490.253292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_no_route -r 2 [ 1490.288878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_no_route -r 2 [ 1490.321273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_no_route -r 2 [ 1490.353070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_no_route -r 2 [ 1490.393458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_no_route -r 2 [ 1490.442666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_no_route -r 2 [ 1490.500757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_no_route -r 2 [ 1490.559756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1490.873053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1490.909929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1490.946639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1490.987561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1491.031346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1491.095070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1491.129293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1491.153821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1491.201864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1491.238431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1491.276869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1491.310634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1491.354793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1491.387609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1492.202336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e local_timeout -r 2 [ 1492.245792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e local_timeout -r 2 [ 1492.285501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e local_timeout -r 2 [ 1492.337464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e local_timeout -r 2 [ 1492.388792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e local_timeout -r 2 [ 1492.436333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e local_timeout -r 2 [ 1492.502789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e local_timeout -r 2 [ 1492.549796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e local_timeout -r 2 [ 1492.612084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1492.645865] LNet: There was an unexpected network error while writing to 192.168.204.155: rc = -22 [ 1492.649514] LNet: Skipped 1 previous similar message [ 1492.929164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1492.963355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1493.002885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1493.063340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1493.110527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1493.162088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1493.207403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1493.233420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1493.251878] LNet: 88983:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1493.257129] LNet: 88983:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1493.279489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1493.311390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1493.343676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1493.407270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1493.456761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1493.497693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1494.048443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp -m GET -e remote_dropped -r 2 [ 1494.086590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.155@tcp1 -m GET -e remote_dropped -r 2 [ 1494.127823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp -m GET -e remote_dropped -r 2 [ 1494.172594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp -d 192.168.204.55@tcp1 -m GET -e remote_dropped -r 2 [ 1494.205859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp -m GET -e remote_dropped -r 2 [ 1494.257324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.155@tcp1 -m GET -e remote_dropped -r 2 [ 1494.308571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp -m GET -e remote_dropped -r 2 [ 1494.369714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.55@tcp1 -d 192.168.204.55@tcp1 -m GET -e remote_dropped -r 2 [ 1494.430673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.55@tcp 192.168.204.155@tcp [ 1494.453149] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.155@tcp [ 1494.698714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1494.728572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1494.786199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1494.819829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1494.843111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1494.871902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1494.908168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1494.934426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1494.960103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1494.984350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1495.029221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1495.088944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1495.169147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1495.229565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1501.250349] LNet: 89181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.257969] LNetError: 89181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.264484] LNet: Removed LNI 192.168.204.55@tcp [ 1501.292467] LNet: Removed LNI 192.168.204.55@tcp1 [ 1501.683674] Key type .llcrypt unregistered [ 1501.688446] Key type ._llcrypt unregistered [ 1519.508358] Key type ._llcrypt registered [ 1519.510706] Key type .llcrypt registered [ 1520.773085] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:38:34 (1786567114) [ 1521.575831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.593415] alg: No test for adler32 (adler32-zlib) [ 1522.605425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.687193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.758508] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1523.642093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1523.683618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.55@tcp [ 1523.705819] LNetError: 90160:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.55@tcp to recovery [ 1523.836636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.55@tcp [ 1523.986021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1530.709737] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:38:44 (1786567124) [ 1531.109076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.155853] LNet: 90750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.177672] LNetError: 90750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.217148] LNet: Removed LNI 192.168.204.55@tcp [ 1531.246434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.310964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.366469] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1532.188953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1532.212226] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1533.282155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.55@tcp [ 1533.308840] LNetError: 91017:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.55@tcp to recovery [ 1533.344434] LNet: 1 local NIs in recovery (showing 1): 192.168.204.55@tcp [ 1533.349274] LNet: There was an unexpected network error while writing to 192.168.204.55: rc = -22 [ 1533.553276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.55@tcp [ 1543.341899] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:38:57 (1786567137) [ 1545.492391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg455-server [ 1547.379553] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:39:00 (1786567140) [ 1549.507992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg455-server [ 1551.864514] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:39:05 (1786567145) [ 1553.997417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg455-server [ 1555.980342] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:39:09 (1786567149) [ 1556.177234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.225155] LNet: 92639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.241328] LNetError: 92639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.257452] LNet: Removed LNI 192.168.204.55@tcp [ 1556.267238] LNet: Removed LNI 192.168.204.55@tcp1 [ 1556.285204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.356786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1556.396593] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1557.211879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1557.297953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1557.345762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1557.431724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1557.483556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1557.549378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1564.405959] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:39:17 (1786567157) [ 1564.522640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.553180] LNet: 93361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.565105] LNetError: 93361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.571428] LNetError: 93361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1564.600477] LNet: Removed LNI 192.168.204.55@tcp [ 1564.645528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.700642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.726835] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1573.398595] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:39:26 (1786567166) [ 1573.706640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.722383] LNet: 94069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.726944] LNetError: 94069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.740495] LNet: Removed LNI 192.168.204.55@tcp [ 1573.771542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.839616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.863581] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1582.957375] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:39:35 (1786567175) [ 1584.765694] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1587.103532] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:39:40 (1786567180) [ 1587.731482] LNet: 95105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.738405] LNetError: 95105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.751627] LNet: Removed LNI 192.168.204.55@tcp [ 1588.626685] Key type .llcrypt unregistered [ 1588.629965] Key type ._llcrypt unregistered [ 1595.497913] Key type ._llcrypt registered [ 1595.501209] Key type .llcrypt registered [ 1595.682475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1595.699639] alg: No test for adler32 (adler32-zlib) [ 1596.708753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1596.757591] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1605.057315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1610.039680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.155@tcp [ 1610.520923] LNet: 95531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.532954] LNetError: 95531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.549948] LNet: Removed LNI 192.168.204.55@tcp [ 1611.170583] Key type .llcrypt unregistered [ 1611.172379] Key type ._llcrypt unregistered [ 1624.444177] Key type ._llcrypt registered [ 1624.446496] Key type .llcrypt registered [ 1625.793796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1627.863929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1629.857193] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:40:23 (1786567223) [ 1631.536601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1633.731355] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:40:26 (1786567226) [ 1636.171439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg455-server [ 1638.017939] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:40:31 (1786567231) [ 1640.490979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg455-server [ 1642.750625] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:40:36 (1786567236) [ 1644.002752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1644.161853] alg: No test for adler32 (adler32-zlib) [ 1645.361834] Lustre: Lustre: Build Version: 2.17.57_1_gc4b410f [ 1645.620230] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1647.319436] Key type lgssc registered [ 1648.537560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1671.680266] Key type lgssc unregistered [ 1672.103160] LNet: 98712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.113749] LNetError: 98712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.138169] LNet: Removed LNI 192.168.204.55@tcp [ 1673.175584] Key type .llcrypt unregistered [ 1673.184771] Key type ._llcrypt unregistered [ 1689.560449] Key type ._llcrypt registered [ 1689.567000] Key type .llcrypt registered [ 1691.440767] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:41:24 (1786567284) [ 1691.963451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.972684] alg: No test for adler32 (adler32-zlib) [ 1692.941285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.038645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1693.119730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1693.225238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1693.331638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1701.920127] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:41:34 (1786567294) [ 1702.025962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.062854] LNet: 100120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.102631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.215952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.271057] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1703.481758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.507222] LNet: 100272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.520057] LNetError: 100272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.541651] LNet: Removed LNI 192.168.204.55@tcp [ 1703.571174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.619249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1703.638664] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1704.217052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.232755] LNetError: 100300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.257606] LNet: Removed LNI 192.168.204.55@tcp [ 1704.280436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.373826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1704.406203] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1713.033289] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:41:46 (1786567306) [ 1713.285554] LNet: 100888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.292310] LNet: 100888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1713.302225] LNetError: 100888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.327580] LNet: Removed LNI 192.168.204.55@tcp [ 1714.141663] Key type .llcrypt unregistered [ 1714.144298] Key type ._llcrypt unregistered [ 1721.717370] Key type ._llcrypt registered [ 1721.729823] Key type .llcrypt registered [ 1721.932741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1721.956406] alg: No test for adler32 (adler32-zlib) [ 1722.905888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.064962] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1723.069933] LNet: *** cfs_fail_loc=e003, val=100*** [ 1723.874909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1723.894333] LNet: *** cfs_fail_loc=e003, val=100*** [ 1723.897441] LNet: Skipped 2 previous similar messages [ 1723.904547] Ping reply received with mapping: [ 1723.909310] found metadata reply NI[192.168.204.55@tcp] [ 1723.912626] 37 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1723.917708] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1723.923791] 5a 5a 5a 5a 5a 5a 5a 5a [ 1724.075595] LNet: 101293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.088026] LNetError: 101293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.101945] LNet: Removed LNI 192.168.204.55@tcp [ 1724.881398] Key type .llcrypt unregistered [ 1724.885398] Key type ._llcrypt unregistered [ 1738.425742] Key type ._llcrypt registered [ 1738.432276] Key type .llcrypt registered [ 1740.859659] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:42:13 (1786567333) [ 1741.950296] Key type .llcrypt unregistered [ 1741.960995] Key type ._llcrypt unregistered [ 1752.014208] Key type ._llcrypt registered [ 1752.020347] Key type .llcrypt registered [ 1752.220577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1752.233943] alg: No test for adler32 (adler32-zlib) [ 1753.374517] LNetError: 102352:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1754.034505] Key type .llcrypt unregistered [ 1754.036990] Key type ._llcrypt unregistered [ 1760.211913] Key type ._llcrypt registered [ 1760.216546] Key type .llcrypt registered [ 1762.197527] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:42:35 (1786567355) [ 1763.026573] Key type .llcrypt unregistered [ 1763.031800] Key type ._llcrypt unregistered [ 1770.666625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1770.672724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1772.109376] Key type ._llcrypt registered [ 1772.111921] Key type .llcrypt registered [ 1772.281129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1772.300028] alg: No test for adler32 (adler32-zlib) [ 1773.175754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.242661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1783.381290] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:42:57 (1786567377) [ 1783.650095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.698181] LNet: 104134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.706209] LNetError: 104134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1783.738485] LNet: Removed LNI 10.1.2.3@tcp [ 1783.796861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.983934] Key type .llcrypt unregistered [ 1784.985844] Key type ._llcrypt unregistered [ 1792.990552] Key type ._llcrypt registered [ 1792.993912] Key type .llcrypt registered [ 1793.205922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1793.233477] alg: No test for adler32 (adler32-zlib) [ 1794.218995] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1794.309141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1795.557643] LNet: 104437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.573739] LNetError: 104437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.645725] LNet: Removed LNI 192.168.204.55@tcp [ 1797.358447] Key type .llcrypt unregistered [ 1797.363566] Key type ._llcrypt unregistered [ 1811.145933] Key type ._llcrypt registered [ 1811.151500] Key type .llcrypt registered [ 1812.739271] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:43:25 (1786567405) [ 1813.365372] Key type .llcrypt unregistered [ 1813.370385] Key type ._llcrypt unregistered [ 1820.453817] Key type ._llcrypt registered [ 1820.457918] Key type .llcrypt registered [ 1820.771207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1820.805526] alg: No test for adler32 (adler32-zlib) [ 1828.908143] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:43:42 (1786567422) [ 1829.347644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.415986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.527009] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.552722] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1830.576663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.626568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1830.679574] LNet: Removed LNI 192.168.204.55@tcp [ 1830.970274] LNet: 106197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.519427] Key type .llcrypt unregistered [ 1831.522238] Key type ._llcrypt unregistered [ 1843.379910] Key type ._llcrypt registered [ 1843.383689] Key type .llcrypt registered [ 1844.912948] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:43:58 (1786567438) [ 1845.334408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.352615] alg: No test for adler32 (adler32-zlib) [ 1846.369793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.458540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.494976] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1847.259943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1847.348743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.55@tcp [ 1847.380710] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1847.384952] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.398234] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1847, 0, 0 [ 1847.406254] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1847.410316] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1847.415177] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.421143] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1847, 0, 0 [ 1847.431859] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1847.434776] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1847.441109] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.443474] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1847, 0, 1897 [ 1847.448663] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1847.450299] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1847.454533] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.459909] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1847, 0, 1897 [ 1847.464199] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1855.693658] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:44:09 (1786567449) [ 1856.120840] LNet: 107739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.129049] LNetError: 107739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.193760] LNet: Removed LNI 192.168.204.55@tcp [ 1857.850571] Key type .llcrypt unregistered [ 1857.852837] Key type ._llcrypt unregistered [ 1865.024041] Key type ._llcrypt registered [ 1865.025623] Key type .llcrypt registered [ 1865.326157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.353575] alg: No test for adler32 (adler32-zlib) [ 1866.310768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1866.336250] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1875.342840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 1887.885215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.952663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1887.996957] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 1889.216340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1889.431953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1889.435594] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1889.441269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1891.824697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1891.846632] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1905.084069] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:44:58 (1786567498) [ 1905.166821] LNet: 109246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.172375] LNetError: 109246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.184833] LNet: Removed LNI 192.168.204.55@tcp [ 1906.207448] LNet: Removed LNI 192.168.204.55@tcp1 [ 1906.216777] LNetError: 109246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1906.227608] LNetError: 109246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1906.249470] LNet: Removed LNI 10.1.2.3@tcp99 [ 1907.104403] Key type .llcrypt unregistered [ 1907.109679] Key type ._llcrypt unregistered [ 1915.031609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1915.035764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1917.886682] Key type ._llcrypt registered [ 1917.893218] Key type .llcrypt registered [ 1918.232284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1918.279573] alg: No test for adler32 (adler32-zlib) [ 1919.302523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.405604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.432796] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1920.156779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1920.175655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1923.264542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1923.364879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1923.434213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1923.453092] LNetError: 110004:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1923.485116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1923.519671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 1924.074652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1924.186530] LNet: 110024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.195928] LNetError: 110024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1924.205180] LNet: Removed LNI 10.1.2.3@tcp [ 1925.291528] LNet: Removed LNI 192.168.204.55@tcp [ 1926.386894] Key type .llcrypt unregistered [ 1926.392673] Key type ._llcrypt unregistered [ 1942.470478] Key type ._llcrypt registered [ 1942.473469] Key type .llcrypt registered [ 1944.062754] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:45:37 (1786567537) [ 1944.501980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.530199] alg: No test for adler32 (adler32-zlib) [ 1945.484703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.562325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.610382] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1946.516550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 1946.614943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg455-client@tcp [ 1953.947809] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:45:47 (1786567547) [ 1954.308366] LNet: 111563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.337157] LNetError: 111563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.360392] LNet: Removed LNI 192.168.204.55@tcp [ 1955.300416] Key type .llcrypt unregistered [ 1955.302363] Key type ._llcrypt unregistered [ 1963.336109] Key type ._llcrypt registered [ 1963.343106] Key type .llcrypt registered [ 1963.613997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.645323] alg: No test for adler32 (adler32-zlib) [ 1964.580699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1964.611678] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1964.670987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1972.865993] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:46:06 (1786567566) [ 1974.888573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1977.377797] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:46:10 (1786567570) [ 1979.350683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1981.512965] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:46:14 (1786567574) [ 1981.703110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.749051] LNet: 112984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.758182] LNetError: 112984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.800556] LNet: Removed LNI 192.168.204.55@tcp [ 1981.854093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.971648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.009883] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 1982.927332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1983.015629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1991.459022] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:46:24 (1786567584) [ 1991.570505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.622242] LNet: 113690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.638494] LNetError: 113690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.701053] LNet: Removed LNI 192.168.204.55@tcp [ 1991.737821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.793945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1991.833528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1992.692149] Key type .llcrypt unregistered [ 1992.695102] Key type ._llcrypt unregistered [ 2006.812555] Key type ._llcrypt registered [ 2006.816176] Key type .llcrypt registered [ 2008.424464] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:46:41 (1786567601) [ 2009.003614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.023191] alg: No test for adler32 (adler32-zlib) [ 2009.996065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.071964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2010.100367] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2011.007982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2011.067334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2018.159555] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:46:51 (1786567611) [ 2018.324570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.350147] LNet: 115254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.356632] LNetError: 115254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.375411] LNet: Removed LNI 192.168.204.55@tcp [ 2018.413170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.485565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2018.513282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2026.564682] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:46:59 (1786567619) [ 2026.883135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2026.925325] LNet: 115839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.959163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.053209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2027.636753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2027.679228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2027.732732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2028.414878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2028.477246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2028.584448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.612653] LNet: 115872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.648873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.740833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2029.479429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2029.571476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2029.679897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2030.100665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2030.188634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2038.898991] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:47:11 (1786567631) [ 2039.038346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.124824] LNet: 116466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.171241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.243605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2039.303288] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2040.122953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2040.160909] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2041.321369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2041.388330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2041.667493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.718271] LNetError: 116752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.729357] LNet: Removed LNI 192.168.204.55@tcp [ 2041.752149] LNet: Removed LNI 192.168.204.55@tcp2 [ 2041.781736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.850262] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2041.857044] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2041.861166] LNet: 116758:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2042.606435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2042.828456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2042.846481] LNetError: 116795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.853812] LNetError: 116795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2042.862817] LNet: Removed LNI 192.168.204.55@tcp [ 2042.878774] LNet: Removed LNI 192.168.204.55@tcp2 [ 2042.900826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.973319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2043.010875] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2043.902739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2043.919872] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2045.353376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2045.403444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2045.456844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2045.708376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.752192] LNet: 117080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.759326] LNet: 117080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2045.775721] LNetError: 117080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.784829] LNetError: 117080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2045.794453] LNet: Removed LNI 192.168.204.55@tcp [ 2046.822081] LNet: Removed LNI 192.168.204.55@tcp2 [ 2046.846331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.966656] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2046.971478] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2046.986279] LNet: 117086:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2047.834415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2048.238982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.278669] LNet: Removed LNI 192.168.204.55@tcp [ 2049.332202] LNet: Removed LNI 192.168.204.55@tcp2 [ 2049.408848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.523764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.562912] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2050.666346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2050.702200] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2052.028383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2052.078755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2052.332377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.373622] LNetError: 117403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.385325] LNetError: 117403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2052.394551] LNet: Removed LNI 192.168.204.55@tcp [ 2052.425681] LNet: Removed LNI 192.168.204.55@tcp2 [ 2052.457743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.519428] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2052.522938] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2052.527558] LNet: 117409:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2053.026222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2053.246203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.303187] LNet: Removed LNI 192.168.204.55@tcp [ 2054.330584] LNet: Removed LNI 192.168.204.55@tcp2 [ 2054.390791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.484622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.520403] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2055.574645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2055.593106] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2056.611586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2056.653345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2056.751331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2056.987926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.015155] LNet: 117731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.023085] LNet: 117731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2058.079215] LNet: Removed LNI 192.168.204.55@tcp [ 2059.110761] LNet: Removed LNI 192.168.204.55@tcp2 [ 2059.195047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.334319] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2059.343296] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2059.350249] LNet: 117737:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2059.909692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2060.224648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.279751] LNet: Removed LNI 192.168.204.55@tcp [ 2061.360726] LNet: Removed LNI 192.168.204.55@tcp2 [ 2061.470761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.577806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.641858] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2062.601696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2062.617538] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2063.663961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2063.704598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2063.801468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2064.275516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.314385] LNetError: 118071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.324540] LNetError: 118071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2064.338443] LNet: Removed LNI 192.168.204.55@tcp [ 2064.363694] LNet: Removed LNI 192.168.204.55@tcp2 [ 2064.398303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.564565] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2064.585424] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2064.617278] LNet: 118077:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2065.538240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2065.975192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.039449] LNet: Removed LNI 192.168.204.55@tcp [ 2067.049893] LNet: Removed LNI 192.168.204.55@tcp2 [ 2067.132064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.241467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.311189] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2068.267920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2068.324790] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2069.382872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2069.419570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2069.462596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2069.534670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2070.015790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.072338] LNet: Removed LNI 192.168.204.55@tcp [ 2072.112280] LNet: Removed LNI 192.168.204.55@tcp2 [ 2072.180040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.277528] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2072.282208] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2072.965763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2081.628992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2083.572514] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:47:57 (1786567677) [ 2083.951183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.981683] LNet: 119197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.989318] LNet: 119197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2083.996908] LNetError: 119197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.002087] LNetError: 119197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2084.007679] LNet: Removed LNI 192.168.204.55@tcp [ 2085.044056] LNet: Removed LNI 192.168.204.55@tcp2 [ 2085.092973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.395229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2085.419700] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2086.127900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2086.159047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2087.140335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.167320] LNet: Removed LNI 10.1.2.3@tcp [ 2087.179622] LNet: Removed LNI 192.168.204.55@tcp [ 2087.196677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.308472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2087.332609] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2087.337052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2087.943707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2087.971789] LNet: Removed LNI 192.168.204.55@tcp [ 2087.984167] LNet: Removed LNI 10.1.2.3@tcp [ 2095.921910] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:48:09 (1786567689) [ 2096.264205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.320619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.421941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.470442] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2097.378129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2097.445538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.484652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.511843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.547646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.579681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.611587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.659396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.725489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.772761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2097.852232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.55@tcp [ 2098.033455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2106.338511] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:48:19 (1786567699) [ 2106.680917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.752103] LNet: Removed LNI 192.168.204.55@tcp [ 2107.825177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.060223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2108.190787] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2108.206037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2108.223908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2110.807575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2110.837622] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2111.694098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2111.746462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2114.077470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2114.116513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2114.207253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.55@tcp -m PUT -e local_timeout -r 1 [ 2114.313857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2114.434846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2114.439226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2114.812755] LNetError: 120876:0:(lib-move.c:4801:lnet_parse()) 192.168.204.55@tcp, src 192.168.204.55@tcp: Dropping PUT (error -108 looking up sender) [ 2115.488168] LNet: Removed LNI 10.1.2.3@tcp [ 2116.532670] LNet: Removed LNI 192.168.204.55@tcp [ 2117.193623] Key type .llcrypt unregistered [ 2117.198601] Key type ._llcrypt unregistered [ 2134.254806] Key type ._llcrypt registered [ 2134.258116] Key type .llcrypt registered [ 2135.955801] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:48:49 (1786567729) [ 2136.570939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2136.598848] alg: No test for adler32 (adler32-zlib) [ 2137.492457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.801192] LNet: 122191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.823739] Key type .llcrypt unregistered [ 2138.832163] Key type ._llcrypt unregistered [ 2145.715562] Key type ._llcrypt registered [ 2145.720767] Key type .llcrypt registered [ 2145.998049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2146.029799] alg: No test for adler32 (adler32-zlib) [ 2147.060264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2147.087941] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2155.332605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 2155.585582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2156.315590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.155@tcp [ 2156.672404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.724323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2156.779354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2156.851660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2156.914630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2156.975622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2157.041870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2157.093162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2157.122205] LNet: 122645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2157.149992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2157.202208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2157.261118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2157.326653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2157.347456] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.155@tcp [ 2157.377388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2157.422827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2158.336317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2158.622928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2158.650871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2158.677363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2158.707293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2158.739796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2158.784622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2158.820214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2158.861545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2158.879469] LNet: 122762:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.900371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2158.932412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2158.962517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2159.004533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2159.044522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2159.079555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2159.831584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.155@tcp [ 2160.166467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2160.223647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2160.260900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2160.291479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2160.322514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2160.375276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2160.457715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2160.523258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2160.569585] LNet: 122880:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2160.603930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2160.664611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2160.763856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2160.816616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2160.891528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2160.935958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.806195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2162.106532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2162.164183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2162.202331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2162.249560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2162.319232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2162.370792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2162.412961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2162.470772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2162.525623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2162.569373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2162.624463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2162.658374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2162.718674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2162.778758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2163.252116] LNet: 123047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.268242] LNetError: 123047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.282451] LNet: Removed LNI 192.168.204.55@tcp [ 2163.665807] Key type .llcrypt unregistered [ 2163.667502] Key type ._llcrypt unregistered [ 2182.923932] Key type ._llcrypt registered [ 2182.926079] Key type .llcrypt registered [ 2184.447668] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:49:37 (1786567777) [ 2184.880212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.898222] alg: No test for adler32 (adler32-zlib) [ 2185.913630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.439901] LNet: 123935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.231435] Key type .llcrypt unregistered [ 2187.237649] Key type ._llcrypt unregistered [ 2194.613729] Key type ._llcrypt registered [ 2194.616926] Key type .llcrypt registered [ 2194.917249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2194.958606] alg: No test for adler32 (adler32-zlib) [ 2195.904538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2195.928330] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2204.612884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.155@tcp [ 2215.226837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.286791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2215.320959] LNet: Added LNI 192.168.204.55@tcp1 [8/256/0/180] [ 2216.574430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2217.335994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.155@tcp [ 2217.443980] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.155@tcp [ 2217.670294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2217.698385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2217.743754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2217.808686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2217.871746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2217.915420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2217.950345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2217.990456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2218.009153] LNet: 124592:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.036919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2218.067895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2218.115659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2218.154840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2218.215100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2218.263460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.381783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2219.722388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.786937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2219.833943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.904585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.952786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.987777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.014109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.049718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.069219] LNet: 124709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.098643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2220.155727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2220.193667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2220.232601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2220.272195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2220.320150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.207619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.155@tcp [ 2221.481806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.512582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2221.547350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.588345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.613698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.651308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2221.682110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2221.721305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2221.759894] LNet: 124826:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.815690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.871872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.932661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.983182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.030920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.059853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.922871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2223.229602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.283387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2223.307780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.353434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.390105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.417350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2223.460575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2223.504553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2223.541936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2223.579741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2223.642515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2223.685112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2223.747380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2223.819952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2229.802266] LNet: 125028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.814520] LNetError: 125028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.824502] LNet: Removed LNI 192.168.204.55@tcp [ 2229.834938] LNet: Removed LNI 192.168.204.55@tcp1 [ 2230.325961] Key type .llcrypt unregistered [ 2230.329593] Key type ._llcrypt unregistered [ 2250.720517] Key type ._llcrypt registered [ 2250.724215] Key type .llcrypt registered [ 2252.213563] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:50:45 (1786567845) [ 2252.675157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.686333] alg: No test for adler32 (adler32-zlib) [ 2253.617552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.657697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.682334] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2254.576641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2254.608336] LNet: Added LNI 192.168.204.55@tcp2 [8/256/0/180] [ 2256.004352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.55@tcp [ 2256.174632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2257.247224] LNet: Removed LNI 192.168.204.55@tcp [ 2267.217767] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:51:00 (1786567860) [ 2269.065674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2270.894817] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:51:04 (1786567864) [ 2279.934158] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:51:13 (1786567873) [ 2289.597869] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:51:23 (1786567883) [ 2298.053369] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:51:31 (1786567891) [ 2305.762453] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:51:39 (1786567899) [ 2314.598865] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:51:48 (1786567908) [ 2322.934126] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:51:56 (1786567916) [ 2331.232920] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:52:04 (1786567924) [ 2339.653299] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:52:12 (1786567932) [ 2347.899382] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:52:21 (1786567941) [ 2355.798617] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:52:29 (1786567949) [ 2364.475445] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:52:37 (1786567957) [ 2371.523763] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:52:45 (1786567965) [ 2377.680584] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:52:51 (1786567971) [ 2384.967926] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:52:58 (1786567978) [ 2393.263749] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:53:06 (1786567986) [ 2401.633447] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:53:15 (1786567995) [ 2409.506259] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:53:23 (1786568003) [ 2409.932238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.972059] LNet: 136727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.983626] LNetError: 136727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.004603] LNet: Removed LNI 192.168.204.55@tcp2 [ 2410.038392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.161859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.183617] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2411.421393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.457990] LNet: 136869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.466980] LNetError: 136869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.485937] LNet: Removed LNI 192.168.204.55@tcp [ 2411.517693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.642834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2411.684974] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2419.494772] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:53:32 (1786568012) [ 2427.568258] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:53:41 (1786568021) [ 2435.128963] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:53:48 (1786568028) [ 2443.243608] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:53:56 (1786568036) [ 2450.730270] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:54:04 (1786568044) [ 2459.482788] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:54:12 (1786568052) [ 2467.858176] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:54:21 (1786568061) [ 2477.340063] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:54:31 (1786568071) [ 2486.939178] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:54:40 (1786568080) [ 2494.308620] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:54:48 (1786568088) [ 2501.471735] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:54:55 (1786568095) [ 2508.419736] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:55:02 (1786568102) [ 2515.519922] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2276 sec ========== 16:55:09 (1786568109) [ 2515.966302] LNet: 144338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2515.971910] LNetError: 144338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.987314] LNet: Removed LNI 192.168.204.55@tcp [ 2516.662934] Key type .llcrypt unregistered [ 2516.665069] Key type ._llcrypt unregistered [ 2528.965390] Key type ._llcrypt registered [ 2528.967147] Key type .llcrypt registered [ 2529.369811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2529.387221] alg: No test for adler32 (adler32-zlib) [ 2530.558403] Lustre: Lustre: Build Version: 2.17.57_1_gc4b410f [ 2530.915392] LNet: Added LNI 192.168.204.55@tcp [8/256/0/180] [ 2532.647177] Key type lgssc registered [ 2533.759752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2594.166994] Lustre: Mounted lustre-client - version 2.17.57_1_gc4b410f [ 2599.092611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2607.144761] Lustre: Unmounted lustre-client [ 2634.306064] Key type lgssc unregistered [ 2634.633273] LNet: 147297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.640344] LNetError: 147297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2634.654502] LNet: Removed LNI 192.168.204.55@tcp [ 2635.532167] Key type .llcrypt unregistered [ 2635.535638] Key type ._llcrypt unregistered