[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 427797911 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003169] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007859] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011014] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.021799] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026341] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.031107] *** VALIDATE proc *** [ 0.032244] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038011] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041847] debug: unmapping init [mem 0xffffffff9c859000-0xffffffff9c860fff] [ 0.043157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044639] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052224] rcu: Hierarchical SRCU implementation. [ 0.054437] smp: Bringing up secondary CPUs ... [ 0.055601] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.059427] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.267815] node 0 deferred pages initialised in 203ms [ 0.271011] devtmpfs: initialized [ 0.272221] x86/mm: Memory block size: 128MB [ 0.274729] gcov: version magic: 0x41383552 [ 0.276316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.280144] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.283520] pinctrl core: initialized pinctrl subsystem [ 0.285217] [ 0.285982] ************************************************************* [ 0.289017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292018] ** ** [ 0.294018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.296018] ** ** [ 0.299022] ** This means that this kernel is built to expose internal ** [ 0.302036] ** IOMMU data structures, which may compromise security on ** [ 0.305019] ** your system. ** [ 0.307017] ** ** [ 0.311023] ** If you see this message and you are not debugging the ** [ 0.314017] ** kernel, report this immediately to your vendor! ** [ 0.316015] ** ** [ 0.319020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.321019] ************************************************************* [ 0.324867] NET: Registered protocol family 16 [ 0.327515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.330091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.333089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.338166] cpuidle: using governor menu [ 0.339688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.342516] PCI: Using configuration type 1 for base access [ 0.345157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.356094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.358064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.362129] cryptd: max_cpu_qlen set to 1000 [ 0.366241] ACPI: Added _OSI(Module Device) [ 0.367027] ACPI: Added _OSI(Processor Device) [ 0.369028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.371052] ACPI: Added _OSI(Processor Aggregator Device) [ 0.375000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.381105] ACPI: Interpreter enabled [ 0.383048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.384012] ACPI: Using IOAPIC for interrupt routing [ 0.386119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.390382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.402230] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.404041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.407020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.411080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.416373] acpiphp: Slot [2] registered [ 0.418147] acpiphp: Slot [5] registered [ 0.419143] acpiphp: Slot [6] registered [ 0.421174] acpiphp: Slot [3] registered [ 0.423141] acpiphp: Slot [4] registered [ 0.424147] acpiphp: Slot [7] registered [ 0.426154] acpiphp: Slot [8] registered [ 0.428127] acpiphp: Slot [9] registered [ 0.429109] acpiphp: Slot [10] registered [ 0.431171] acpiphp: Slot [11] registered [ 0.433176] acpiphp: Slot [12] registered [ 0.434128] acpiphp: Slot [13] registered [ 0.436142] acpiphp: Slot [14] registered [ 0.438154] acpiphp: Slot [15] registered [ 0.439215] acpiphp: Slot [16] registered [ 0.441128] acpiphp: Slot [17] registered [ 0.442183] acpiphp: Slot [18] registered [ 0.444159] acpiphp: Slot [19] registered [ 0.446108] acpiphp: Slot [20] registered [ 0.448124] acpiphp: Slot [21] registered [ 0.450107] acpiphp: Slot [22] registered [ 0.452106] acpiphp: Slot [23] registered [ 0.453228] acpiphp: Slot [24] registered [ 0.455098] acpiphp: Slot [25] registered [ 0.457112] acpiphp: Slot [26] registered [ 0.458161] acpiphp: Slot [27] registered [ 0.460081] acpiphp: Slot [28] registered [ 0.462115] acpiphp: Slot [29] registered [ 0.464127] acpiphp: Slot [30] registered [ 0.465240] acpiphp: Slot [31] registered [ 0.467065] PCI host bridge to bus 0000:00 [ 0.468014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.471023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.473024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.476033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.479040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.482049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.484269] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.487145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.491384] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.498647] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.504065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.507032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.510041] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.513030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.516574] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.518698] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.521050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.524840] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.530014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.540736] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.546013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.553072] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.562022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.568022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.592019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.601707] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.612015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.619015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.633017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.642319] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.644574] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.647418] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.650443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.653290] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.660035] iommu: Default domain type: Passthrough [ 0.662495] SCSI subsystem initialized [ 0.663152] ACPI: bus type USB registered [ 0.665099] usbcore: registered new interface driver usbfs [ 0.667086] usbcore: registered new interface driver hub [ 0.669089] usbcore: registered new device driver usb [ 0.671170] pps_core: LinuxPPS API ver. 1 registered [ 0.672009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676059] PTP clock support registered [ 0.678145] EDAC MC: Ver: 3.0.0 [ 0.680027] PCI: Using ACPI for IRQ routing [ 0.681932] NetLabel: Initializing [ 0.682013] NetLabel: domain hash size = 128 [ 0.683014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684153] NetLabel: unlabeled traffic allowed by default [ 0.686196] vgaarb: loaded [ 0.688261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.690017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.695965] clocksource: Switched to clocksource kvm-clock [ 0.805366] VFS: Disk quotas dquot_6.6.0 [ 0.807213] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.810287] *** VALIDATE ramfs *** [ 0.811753] *** VALIDATE hugetlbfs *** [ 0.813743] pnp: PnP ACPI init [ 0.816161] pnp: PnP ACPI: found 6 devices [ 0.833872] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.839372] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.841758] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.844315] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.847055] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.849735] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.853155] NET: Registered protocol family 2 [ 0.856191] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.861579] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.865611] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.871283] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.875144] TCP: Hash tables configured (established 65536 bind 65536) [ 0.878495] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.881994] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885258] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888644] NET: Registered protocol family 1 [ 0.891702] RPC: Registered named UNIX socket transport module. [ 0.894367] RPC: Registered udp transport module. [ 0.896362] RPC: Registered tcp transport module. [ 0.898375] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901339] NET: Registered protocol family 44 [ 0.902588] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904590] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.907103] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.909637] PCI: CLS 0 bytes, default 64 [ 0.911064] Unpacking initramfs... [ 2.325163] debug: unmapping init [mem 0xffff8becfcc64000-0xffff8becfffcffff] [ 2.331769] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334089] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.337208] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.843491] Initialise system trusted keyrings [ 2.845591] Key type blacklist registered [ 2.848517] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.859159] zbud: loaded [ 2.863258] *** VALIDATE nfs *** [ 2.864533] *** VALIDATE nfs4 *** [ 2.867051] pstore: using deflate compression [ 2.871498] Platform Keyring initialized [ 2.978072] NET: Registered protocol family 38 [ 2.980014] Key type asymmetric registered [ 2.981802] Asymmetric key parser 'x509' registered [ 2.983662] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.987017] io scheduler mq-deadline registered [ 2.988618] io scheduler kyber registered [ 2.989864] io scheduler bfq registered [ 2.991509] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.994818] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.997853] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.001060] ACPI: Power Button [PWRF] [ 3.007029] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.015139] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.027958] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.055678] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.085469] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.090748] Non-volatile memory driver v1.3 [ 3.092690] Linux agpgart interface v0.103 [ 3.123496] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.126831] vda: detected capacity change from 0 to 76779520 [ 3.141621] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.144853] vdb: detected capacity change from 0 to 1073741824 [ 3.151803] libphy: Fixed MDIO Bus: probed [ 3.157140] usbcore: registered new interface driver usbserial_generic [ 3.160091] usbserial: USB Serial support registered for generic [ 3.162826] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.166953] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.168486] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.170801] mousedev: PS/2 mouse device common for all mice [ 3.173583] rtc_cmos 00:05: RTC can wake from S4 [ 3.176135] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.176546] rtc_cmos 00:05: registered as rtc0 [ 3.180658] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.183990] intel_pstate: CPU model not supported [ 3.188623] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.192578] hid: raw HID events driver (C) Jiri Kosina [ 3.195302] usbcore: registered new interface driver usbhid [ 3.195438] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.197145] usbhid: USB HID core driver [ 3.201881] drop_monitor: Initializing network drop monitor service [ 3.203785] Initializing XFRM netlink socket [ 3.205187] NET: Registered protocol family 10 [ 3.207981] Segment Routing with IPv6 [ 3.210737] NET: Registered protocol family 17 [ 3.212572] mpls_gso: MPLS GSO support [ 3.217528] RAS: Correctable Errors collector initialized. [ 3.219631] AVX version of gcm_enc/dec engaged. [ 3.224854] AES CTR mode by8 optimization enabled [ 3.303240] sched_clock: Marking stable (3303131883, 0)->(4182665868, -879533985) [ 3.306710] registered taskstats version 1 [ 3.309490] Loading compiled-in X.509 certificates [ 3.311681] zswap: loaded using pool lzo/zbud [ 3.338376] Key type big_key registered [ 3.351266] Key type encrypted registered [ 3.353297] ima: No TPM chip found, activating TPM-bypass! [ 3.355569] ima: Allocated hash algorithm: sha1 [ 3.357444] ima: No architecture policies found [ 3.359292] evm: Initialising EVM extended attributes: [ 3.361471] evm: security.selinux [ 3.362726] evm: security.ima [ 3.364015] evm: security.capability [ 3.365167] evm: HMAC attrs: 0x1 [ 3.367073] rtc_cmos 00:05: setting system clock to 2026-09-16 13:20:15 UTC (1789564815) [ 3.373275] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.376370] debug: unmapping init [mem 0xffffffff9c582000-0xffffffff9c858fff] [ 3.385092] Write protecting the kernel read-only data: 28672k [ 3.388032] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.390621] debug: unmapping init [mem 0xffffffff9b514000-0xffffffff9b5fffff] [ 3.423102] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.431878] systemd[1]: Detected virtualization kvm. [ 3.433803] systemd[1]: Detected architecture x86-64. [ 3.435822] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.461349] systemd[1]: No hostname configured. [ 3.463155] systemd[1]: Set hostname to . [ 3.465480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467877] systemd[1]: Initializing machine ID from random generator. [ 3.517106] random: ln: uninitialized urandom read (6 bytes read) [ 3.587548] random: systemd: uninitialized urandom read (16 bytes read) [ 3.590878] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.596408] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.601599] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.259207] device-mapper: uevent: version 1.0.3 [ 4.261835] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.044748] virtio_net virtio0 ens2: renamed from eth0 [ 5.122086] scsi host0: ata_piix [ 5.173798] scsi host1: ata_piix [ 5.228559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.231408] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.560313] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.869732] random: crng init done [ 9.871347] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.357777] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.513575] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.765719] SELinux: Disabled at runtime. [ 11.832177] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.841362] systemd[1]: Detected virtualization kvm. [ 11.843318] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.421327] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.425530] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.431259] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.435700] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.439319] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.448363] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.459131] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ 12.604754] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.973823] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.356630] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.359478] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.477226] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.495310] EDAC sbridge: Ver: 1.1.2 [ 14.693199] Key type dns_resolver registered [ 15.011098] NFS: Registering the id_resolver key type [ 15.013345] Key type id_resolver registered [ 15.015068] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-client login: [ 61.660289] libcfs: loading out-of-tree module taints kernel. [ 61.688545] Key type ._llcrypt registered [ 61.691079] Key type .llcrypt registered [ 62.356737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 62.368467] alg: No test for adler32 (adler32-zlib) [ 63.830181] Lustre: Lustre: Build Version: 2.17.58_39_gf62a9ec [ 64.615079] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 66.384874] Key type lgssc registered [ 68.036909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.514007] hrtimer: interrupt took 4054671 ns [ 189.455086] Lustre: Mounted lustre-client - version 2.17.58_39_gf62a9ec [ 195.204982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 208.896873] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing check_logdir /tmp/testlogs/ [ 214.050703] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing yml_node [ 215.012349] Lustre: lustre-OST0000-osc-ffff8bed43e44000: disconnect after 23s idle [ 217.801895] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 219.942906] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 222.216746] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 223.900529] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 16 09:23:54 EDT 2026 [ 241.735364] Lustre: Unmounted lustre-client [ 273.547298] Key type lgssc unregistered [ 273.901601] LNet: 8410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.908402] LNetError: 8410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 273.951049] LNet: Removed LNI 192.168.203.56@tcp [ 274.771140] Key type .llcrypt unregistered [ 274.772884] Key type ._llcrypt unregistered [ 299.017479] Key type ._llcrypt registered [ 299.020308] Key type .llcrypt registered [ 299.197149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 299.210728] alg: No test for adler32 (adler32-zlib) [ 300.099538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 300.159290] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 300.188557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 304.032839] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 304.209980] LNet: 9284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.224548] LNetError: 9284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 304.298156] LNet: Removed LNI 192.168.203.56@tcp [ 305.028399] Key type .llcrypt unregistered [ 305.033686] Key type ._llcrypt unregistered [ 311.223169] Key type ._llcrypt registered [ 311.224852] Key type .llcrypt registered [ 312.405389] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:25:23 (1789565123) [ 312.794953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 312.812139] alg: No test for adler32 (adler32-zlib) [ 313.690246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.796620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.821819] LNet: 9697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.839297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.895557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 313.915927] LNet: 9704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 320.085945] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:25:30 (1789565130) [ 320.161947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.203906] LNet: 10274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.240891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.364451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 325.542957] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:25:36 (1789565136) [ 325.647576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.675896] LNet: 10866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.709210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.809353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 331.510800] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:25:42 (1789565142) [ 331.752521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.779515] LNet: 11450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.811564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.923966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 337.539714] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:25:48 (1789565148) [ 337.765685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.789484] LNet: 12034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.808401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.884865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 338.394782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 344.733347] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:25:55 (1789565155) [ 344.919441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.976531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.067233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.625604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.958985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 351.727592] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:26:02 (1789565162) [ 351.824131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.865267] LNet: 13214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.869780] LNet: 13214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 351.895625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.986933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 358.955273] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:26:09 (1789565169) [ 359.104568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.159826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.250293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 359.287775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.580468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.608462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.944649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 360.021330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.706847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 360.763311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.121183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 361.153861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.854770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 361.978571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.366495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 362.392685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.786354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 362.806941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.036469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 363.060849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.318504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 363.367449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.714463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 363.753274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.004743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 364.031645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.459739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 364.525331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 371.674374] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:26:22 (1789565182) [ 371.768411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.801625] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.804624] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 371.831880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.961036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 372.026882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 380.004701] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:26:30 (1789565190) [ 380.182189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.252432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.380523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 380.440636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 388.334132] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:26:39 (1789565199) [ 388.546878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.593860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.662214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 388.691633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 395.132373] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:26:46 (1789565206) [ 395.238679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.275623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.360324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 395.400872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 401.069603] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:26:51 (1789565211) [ 401.150176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.180714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.254543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 401.316650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 408.187069] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:26:59 (1789565219) [ 408.625859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.648712] LNet: 17420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.661654] LNet: 17420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 408.697355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.768159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.843231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 418.200700] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:27:08 (1789565228) [ 418.835499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.916040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.125395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.567788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.393109] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:27:17 (1789565237) [ 427.442144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.494438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.676911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.853280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.362085] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:27:25 (1789565245) [ 435.551884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.614026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.937193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.293745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.241213] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:27:35 (1789565255) [ 444.379890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.431258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.899762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.117032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.961789] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:27:43 (1789565263) [ 453.171392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.226812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.337053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.395184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.010318] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:27:50 (1789565270) [ 460.103556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.165685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.257376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.311285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.152394] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:27:57 (1789565277) [ 467.446564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.522074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.607974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 474.153920] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:28:04 (1789565284) [ 474.461105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.516363] LNet: 22274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.525578] LNet: 22274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 474.554852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.648991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 483.618919] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:28:13 (1789565293) [ 483.886874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.981921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.145495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 484.718687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 491.155437] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:28:22 (1789565302) [ 491.462893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.538266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.675592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 491.717876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 498.811932] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:28:29 (1789565309) [ 498.885059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.932152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.014578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 499.042543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 506.665391] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:28:36 (1789565316) [ 506.759747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.857455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.002694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 507.056727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 515.255267] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:28:45 (1789565325) [ 515.433874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.542326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.727071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 515.803966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.766189] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:28:56 (1789565336) [ 525.952969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.008125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.126991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 526.231905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 526.270513] LNetError: 25807:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 526.358049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 526.404836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 534.028300] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:29:04 (1789565344) [ 534.237222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.303571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.418333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.d9pUCW4mDw [ 534.519975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.d9pUCW4mDw [ 534.620571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.d9pUCW4mDw [ 542.063031] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:29:12 (1789565352) [ 542.292204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.354904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.432324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 542.488552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 542.544442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.066325] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:29:19 (1789565359) [ 549.222588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.297590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.428515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 549.480233] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 551.141713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 552.422830] Lustre: Lustre: Build Version: 2.17.58_39_gf62a9ec [ 554.360362] Key type lgssc registered [ 555.763363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 559.581550] LNetError: 27567:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.56@tcp99 has no available nets [ 562.333859] Key type lgssc unregistered [ 562.717958] LNetError: 28673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.734457] LNet: Removed LNI 192.168.203.56@tcp [ 563.414689] Key type .llcrypt unregistered [ 563.418099] Key type ._llcrypt unregistered [ 569.982460] Key type ._llcrypt registered [ 569.985678] Key type .llcrypt registered [ 571.154307] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:29:41 (1789565381) [ 571.712524] Key type .llcrypt unregistered [ 571.716273] Key type ._llcrypt unregistered [ 577.348444] Key type ._llcrypt registered [ 577.354369] Key type .llcrypt registered [ 577.604748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.637761] alg: No test for adler32 (adler32-zlib) [ 578.623861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.674350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.691283] LNet: 29582:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 578.697387] LNet: 29582:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 578.704194] LNet: 29582:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 578.718940] LNet: Message forwarding will be enabled in 2s [ 581.792182] LNet: Message forwarding enabled [ 582.171346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 582.216253] LNet: Message forwarding will stop in 2s to 6s [ 585.907599] LNet: Message forwarding disabled [ 586.012771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 586.066176] LNet: 29672:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 586.077755] LNet: 29672:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 586.083679] LNet: 29672:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 586.111658] LNet: Message forwarding will be enabled in 2s [ 586.347405] LNet: 29685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.010696] Key type .llcrypt unregistered [ 587.014539] Key type ._llcrypt unregistered [ 598.622586] Key type ._llcrypt registered [ 598.624830] Key type .llcrypt registered [ 600.099262] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:30:10 (1789565410) [ 600.650917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 600.697535] alg: No test for adler32 (adler32-zlib) [ 601.693628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.867928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 609.288633] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:30:20 (1789565420) [ 609.409319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.428894] LNet: 31079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.445128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.495080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 609.547546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 609.574634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 609.614446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 609.658699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 609.696378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 609.750259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 609.801704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 609.873423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 609.924411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 609.970596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 609.997427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 610.052994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 610.123475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 610.151918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 610.176913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 610.205680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 610.231978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 610.278867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 610.317547] LNetError: 31127:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 617.366757] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:30:28 (1789565428) [ 617.660093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.744348] LNet: 31698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.781938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.870641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 624.293724] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:30:34 (1789565434) [ 626.277726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg356-server [ 628.401852] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:30:39 (1789565439) [ 628.503836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.542383] LNet: 32621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.586735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.673960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.716415] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 629.612486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[57-61]@tcp [ 629.634598] LNet: 32762:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 629.702807] LNetError: 32631:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 629.738198] LNetError: 32631:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 632.811416] LNetError: 32763:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 632.826488] LNetError: 32763:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 636.963021] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:30:47 (1789565447) [ 638.662447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg356-server [ 640.212530] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:30:51 (1789565451) [ 640.341647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.363101] LNet: 33690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.375491] LNetError: 33690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.399854] LNet: Removed LNI 192.168.203.56@tcp [ 640.437313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.526182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.563934] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 641.228793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 641.246804] LNet: 33815:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.256684] LNet: 33815:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 641.286530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 648.203179] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:30:58 (1789565458) [ 648.350429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.370362] LNet: 34407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.377590] LNetError: 34407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.391730] LNet: Removed LNI 192.168.203.56@tcp [ 648.435058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.540150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 648.601740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.658670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.754304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.839965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.889384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.950391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 649.021748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.057386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.141913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.219814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.250750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.301396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 649.384713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.438648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.500835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.584811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.615763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.678836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 649.755238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.798427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.858048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.973337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.018922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.066879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 650.168804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.212354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.252334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 650.328223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.372350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.434719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 650.507695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.571795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.648301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.892694] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:31:08 (1789565468) [ 660.485418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg356-server [ 662.426527] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:31:12 (1789565472) [ 664.179499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg356-server [ 665.730859] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:31:16 (1789565476) [ 665.863821] LNet: 35804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.870985] LNet: 35804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 666.543857] Key type .llcrypt unregistered [ 666.547170] Key type ._llcrypt unregistered [ 673.301043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 673.638107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 673.662985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 673.681271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.284706] Key type ._llcrypt registered [ 675.288681] Key type .llcrypt registered [ 675.665372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.688404] alg: No test for adler32 (adler32-zlib) [ 676.636814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.693052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.737459] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 677.392444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.415597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.487318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 679.535432] LNet: Removed LNI 10.1.2.3@tcp [ 679.634763] LNet: 36520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.642705] LNetError: 36520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.653205] LNet: Removed LNI 192.168.203.56@tcp [ 680.380456] Key type .llcrypt unregistered [ 680.388485] Key type ._llcrypt unregistered [ 692.169699] Key type ._llcrypt registered [ 692.171229] Key type .llcrypt registered [ 693.137600] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:31:44 (1789565504) [ 693.825364] Key type .llcrypt unregistered [ 693.828718] Key type ._llcrypt unregistered [ 700.138334] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.525753] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.541572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.555467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.405493] Key type ._llcrypt registered [ 703.408121] Key type .llcrypt registered [ 703.604749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.624356] alg: No test for adler32 (adler32-zlib) [ 704.509037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.557518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.587824] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 705.396860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.427444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.422584] LNet: 38009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.429476] LNetError: 38009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 708.434416] LNet: Removed LNI 10.1.2.3@tcp [ 708.442716] LNet: Removed LNI 192.168.203.56@tcp [ 709.067526] Key type .llcrypt unregistered [ 709.069347] Key type ._llcrypt unregistered [ 720.338407] Key type ._llcrypt registered [ 720.345336] Key type .llcrypt registered [ 721.421532] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:32:12 (1789565532) [ 721.976372] Key type .llcrypt unregistered [ 721.982331] Key type ._llcrypt unregistered [ 728.067054] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 728.364730] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 728.386500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 728.402236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.290990] Key type ._llcrypt registered [ 731.294692] Key type .llcrypt registered [ 731.584469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.607593] alg: No test for adler32 (adler32-zlib) [ 732.586326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.677884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 732.727289] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 734.929300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.962404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 736.968973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 736.984871] LNet: Removed LNI 10.1.2.3@tcp [ 736.998685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 737.023219] LNet: Removed LNI 10.1.2.31@tcp [ 737.042877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 737.070851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.848940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 737.866855] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 738.987630] LNet: 39926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.001676] LNetError: 39926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 739.014457] LNet: Removed LNI 10.1.2.31@tcp [ 739.029344] LNet: Removed LNI 10.1.2.3@tcp [ 739.595656] Key type .llcrypt unregistered [ 739.601537] Key type ._llcrypt unregistered [ 751.578741] Key type ._llcrypt registered [ 751.581450] Key type .llcrypt registered [ 753.318407] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:32:43 (1789565563) [ 753.659427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 753.672850] alg: No test for adler32 (adler32-zlib) [ 754.677562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.753104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.782426] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 755.535831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.588301] LNet: 40877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.594830] LNetError: 40877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.612433] LNet: Removed LNI 192.168.203.56@tcp [ 755.646552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.749357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.56@tcp [ 755.779337] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 756.371259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.56@tcp [ 756.419592] LNet: Removed LNI 192.168.203.56@tcp [ 756.474163] LNet: 41012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.046474] Key type .llcrypt unregistered [ 757.050313] Key type ._llcrypt unregistered [ 768.834429] Key type ._llcrypt registered [ 768.836659] Key type .llcrypt registered [ 770.213449] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:33:00 (1789565580) [ 772.426450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg356-server [ 774.038297] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:33:04 (1789565584) [ 774.558262] Key type .llcrypt unregistered [ 774.560319] Key type ._llcrypt unregistered [ 780.971869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.128948] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.140888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.150732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.014237] Key type ._llcrypt registered [ 784.020878] Key type .llcrypt registered [ 784.412981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.445440] alg: No test for adler32 (adler32-zlib) [ 785.404573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.536888] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 786.154413] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 788.679630] LNet: 42845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.699150] LNetError: 42845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.707114] LNet: Removed LNI 192.168.203.56@tcp1 [ 788.724213] LNet: Removed LNI 10.1.2.3@tcp2 [ 789.386357] Key type .llcrypt unregistered [ 789.388231] Key type ._llcrypt unregistered [ 802.875106] Key type ._llcrypt registered [ 802.876880] Key type .llcrypt registered [ 804.047615] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:33:34 (1789565614) [ 804.658267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.676414] alg: No test for adler32 (adler32-zlib) [ 805.598313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.689599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.726215] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 806.710731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.743110] LNet: 43813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.756051] LNetError: 43813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.776212] LNet: Removed LNI 192.168.203.56@tcp [ 806.808898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.938625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 807.006193] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 814.592655] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:33:45 (1789565625) [ 814.754432] LNet: 44397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.764992] LNetError: 44397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.791854] LNet: Removed LNI 192.168.203.56@tcp [ 815.774468] Key type .llcrypt unregistered [ 815.779989] Key type ._llcrypt unregistered [ 822.735705] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.893616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.898762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.905084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 825.447706] Key type ._llcrypt registered [ 825.450775] Key type .llcrypt registered [ 825.669794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.696093] alg: No test for adler32 (adler32-zlib) [ 826.624117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.679747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.700348] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 827.420444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 827.449951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.365961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.383206] LNet: 45145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.388229] LNetError: 45145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.393698] LNet: Removed LNI 10.1.2.3@tcp [ 829.407074] LNet: Removed LNI 192.168.203.56@tcp [ 829.446990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.508634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 829.533953] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 829.538112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.104549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.149395] LNet: 45172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.161099] LNetError: 45172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.169765] LNetError: 45172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 830.184836] LNet: Removed LNI 10.1.2.3@tcp [ 830.199829] LNet: Removed LNI 192.168.203.56@tcp [ 830.221241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.291379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.335212] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 830.344865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.911748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 830.956348] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 832.300419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.329812] LNet: 45335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.344473] LNetError: 45335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.356231] LNetError: 45335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 832.369670] LNet: Removed LNI 10.1.2.3@tcp [ 832.375282] LNet: Removed LNI 192.168.203.56@tcp [ 832.396339] LNet: Removed LNI 192.168.203.56@tcp2 [ 832.436096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.545272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 832.591760] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 832.605903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 832.614197] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 840.456429] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:34:11 (1789565651) [ 840.735457] LNet: 45933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.752869] LNetError: 45933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.756237] LNetError: 45933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 840.774222] LNet: Removed LNI 10.1.2.3@tcp [ 840.784649] LNet: Removed LNI 192.168.203.56@tcp [ 840.799137] LNet: Removed LNI 192.168.203.56@tcp2 [ 841.449178] Key type .llcrypt unregistered [ 841.452266] Key type ._llcrypt unregistered [ 848.513568] Key type ._llcrypt registered [ 848.522582] Key type .llcrypt registered [ 848.791761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.829658] alg: No test for adler32 (adler32-zlib) [ 849.773446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.825071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.862240] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 850.642846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.663598] LNet: 46325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.668874] LNetError: 46325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.688711] LNet: Removed LNI 192.168.203.56@tcp [ 850.722949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.810547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 850.875107] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 851.484436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.513918] LNet: 46355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.529508] LNetError: 46355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.554721] LNet: Removed LNI 192.168.203.56@tcp [ 851.573773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.651373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.697544] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 852.207399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.244240] LNet: Removed LNI 192.168.203.56@tcp [ 852.274507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.360697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 852.397896] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 852.838864] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.988037] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.992870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.012568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.341386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.372050] LNet: 46465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.382550] LNet: 46465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 855.397278] LNetError: 46465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.408887] LNetError: 46465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 855.434285] LNet: Removed LNI 192.168.203.56@tcp [ 855.497987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.648531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 855.701824] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 863.549572] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:34:34 (1789565674) [ 863.989568] LNet: 47063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.996964] LNetError: 47063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.013814] LNet: Removed LNI 192.168.203.56@tcp [ 864.715852] Key type .llcrypt unregistered [ 864.719927] Key type ._llcrypt unregistered [ 870.576442] Key type ._llcrypt registered [ 870.579595] Key type .llcrypt registered [ 870.767568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.780418] alg: No test for adler32 (adler32-zlib) [ 871.695486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.755554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.783645] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 872.551898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.584147] LNet: 47458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.598449] LNetError: 47458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.633052] LNet: Removed LNI 192.168.203.56@tcp [ 872.658241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.739270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 872.763847] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 882.113161] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:34:52 (1789565692) [ 882.255670] LNet: 48048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.264632] LNetError: 48048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.291184] LNet: Removed LNI 192.168.203.56@tcp [ 883.074933] Key type .llcrypt unregistered [ 883.080662] Key type ._llcrypt unregistered [ 889.849511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.057323] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.060557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.063438] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 892.543645] Key type ._llcrypt registered [ 892.545960] Key type .llcrypt registered [ 892.674343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 892.694985] alg: No test for adler32 (adler32-zlib) [ 893.629500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.677686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.697757] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 894.547582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.566624] LNet: 48495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.573876] LNetError: 48495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.587910] LNet: Removed LNI 192.168.203.56@tcp [ 894.612527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.667364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 894.697062] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 901.396945] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:35:12 (1789565712) [ 901.508090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.550782] LNet: 49089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.561933] LNetError: 49089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.605274] LNet: Removed LNI 192.168.203.56@tcp [ 901.647888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.803182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 908.276337] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:35:18 (1789565718) [ 908.348408] LNet: 49669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.953888] Key type .llcrypt unregistered [ 908.956488] Key type ._llcrypt unregistered [ 915.289709] Key type ._llcrypt registered [ 915.291846] Key type .llcrypt registered [ 915.498250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.525298] alg: No test for adler32 (adler32-zlib) [ 916.426221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.469566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.484853] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 917.200992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.227262] LNet: 50072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.232822] LNetError: 50072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.252777] LNet: Removed LNI 192.168.203.56@tcp [ 917.281933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.347633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 917.369889] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 923.969601] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:35:34 (1789565734) [ 924.131074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.166518] LNet: 50660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.184356] LNetError: 50660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.225152] LNet: Removed LNI 192.168.203.56@tcp [ 924.285119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.362291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 931.025450] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:35:41 (1789565741) [ 932.999799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 935.036204] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:35:45 (1789565745) [ 936.718282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 939.201298] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:35:49 (1789565749) [ 941.290980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 943.781478] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:35:53 (1789565753) [ 945.582517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 947.312105] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:35:57 (1789565757) [ 949.004571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 950.846419] Lustre: DEBUG MARKER: == sanity-lnet test 163: Check import of net with mixed IPv6/IPv4 explicit NIDs ========================================================== 09:36:01 (1789565761) [ 951.162794] LNet: 52688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.983139] Key type .llcrypt unregistered [ 951.985379] Key type ._llcrypt unregistered [ 958.601226] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.907231] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.909762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.915230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.507619] Key type ._llcrypt registered [ 961.509770] Key type .llcrypt registered [ 961.727338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.757980] alg: No test for adler32 (adler32-zlib) [ 962.669974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.738508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-163-expected.yaml [ 962.761412] LNet: Added LNI 2001:db8:0:f101::1@tcp [8/256/0/180] [ 962.766846] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 962.856614] LNet: 53022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.863288] LNetError: 53022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.871918] LNet: Removed LNI 2001:db8:0:f101::1@tcp [ 962.904066] LNet: Removed LNI 10.1.2.3@tcp1 [ 963.456439] Key type .llcrypt unregistered [ 963.462678] Key type ._llcrypt unregistered [ 979.639247] Key type ._llcrypt registered [ 979.651615] Key type .llcrypt registered [ 981.380876] Lustre: DEBUG MARKER: == sanity-lnet test 164: Check no issue when reading 'nid' in all scenarios ========================================================== 09:36:31 (1789565791) [ 981.850293] Key type .llcrypt unregistered [ 981.853703] Key type ._llcrypt unregistered [ 989.924058] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 990.184473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 992.671567] Key type ._llcrypt registered [ 992.676650] Key type .llcrypt registered [ 992.868754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.897604] alg: No test for adler32 (adler32-zlib) [ 993.945313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.265367] Lustre: DEBUG MARKER: Check malformed nid fails [ 995.605649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 995.720974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.772654] LNet: 54259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.794297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.603851] Lustre: DEBUG MARKER: Check wrong net in nid fails [ 997.812583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 997.914204] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 999.418798] Lustre: DEBUG MARKER: sanity-lnet test_164: @@@@@@ FAIL: Wrong net imported [ 1010.119848] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:37:00 (1789565820) [ 1010.379016] LNet: 55117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.414357] LNetError: 55117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.455426] LNet: Removed LNI 192.168.203.56@tcp2 [ 1011.165503] Key type .llcrypt unregistered [ 1011.170519] Key type ._llcrypt unregistered [ 1019.293963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.542161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.226425] Key type ._llcrypt registered [ 1022.229748] Key type .llcrypt registered [ 1022.599501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.634964] alg: No test for adler32 (adler32-zlib) [ 1023.644944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.737916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.767710] LNet: Added LNI 192.168.203.56@tcp [8/512/0/180] [ 1023.772773] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1023.780121] LNet: 55449:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.894912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.918749] LNet: 55468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.926890] LNetError: 55468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.932956] LNet: Removed LNI 10.1.2.3@tcp [ 1023.935876] LNetError: 55468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.954111] LNet: Removed LNI 192.168.203.56@tcp [ 1023.975875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.038418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.080514] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1024.186239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.228130] LNet: 55492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.240585] LNetError: 55492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.261050] LNet: Removed LNI 192.168.203.56@tcp [ 1024.294851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.373881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.394085] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1024.397747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.400428] LNet: 55500:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.469738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.490572] LNet: 55518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.495697] LNetError: 55518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.500796] LNet: Removed LNI 10.1.2.3@tcp [ 1024.506312] LNetError: 55518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.517488] LNet: Removed LNI 192.168.203.56@tcp [ 1024.559850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.616325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.634350] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1024.645710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.649447] LNet: 55526:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.863825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.917927] LNet: 55552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.926558] LNetError: 55552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.934283] LNet: Removed LNI 10.1.2.3@tcp [ 1024.942987] LNetError: 55552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.967119] LNet: Removed LNI 192.168.203.56@tcp [ 1024.995578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.112301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1025.169198] LNet: Added LNI 192.168.203.56@tcp [8/256/0/182] [ 1025.173802] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1025.327557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.361337] LNet: 55580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.374732] LNetError: 55580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.381358] LNet: Removed LNI 10.1.2.3@tcp [ 1025.387517] LNetError: 55580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.409184] LNet: Removed LNI 192.168.203.56@tcp [ 1025.453397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.561528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1025.607872] LNet: Added LNI 192.168.203.56@tcp [8/256/0/182] [ 1025.620501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1025.624728] LNet: 55588:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1025.759927] LNet: Removed LNI 10.1.2.3@tcp [ 1025.794481] LNet: Removed LNI 192.168.203.56@tcp [ 1026.570478] Key type .llcrypt unregistered [ 1026.573667] Key type ._llcrypt unregistered [ 1039.218528] Key type ._llcrypt registered [ 1039.224261] Key type .llcrypt registered [ 1040.683917] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:37:31 (1789565851) [ 1042.212400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1044.376621] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:37:34 (1789565854) [ 1044.746410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1044.780673] alg: No test for adler32 (adler32-zlib) [ 1045.706155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.774710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1045.805403] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1046.724059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.56 [ 1046.754381] LNet: 56850:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1046.773899] LNet: Removed LNI 192.168.203.56@tcp [ 1054.771407] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:37:44 (1789565864) [ 1054.928287] LNet: 57411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.944226] LNetError: 57411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.977543] LNet: Removed LNI 192.168.203.56@tcp [ 1055.884340] Key type .llcrypt unregistered [ 1055.887608] Key type ._llcrypt unregistered [ 1062.536959] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1062.821120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1065.715486] Key type ._llcrypt registered [ 1065.723078] Key type .llcrypt registered [ 1066.125672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.161395] alg: No test for adler32 (adler32-zlib) [ 1068.006724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.261937] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:38:06 (1789565886) [ 1076.419250] LNet: 58329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.434220] LNetError: 58329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1076.467266] LNet: Removed LNI 10.1.2.3@tcp [ 1077.099479] Key type .llcrypt unregistered [ 1077.101181] Key type ._llcrypt unregistered [ 1084.962550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1085.279034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1085.288550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1085.293198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1088.083672] Key type ._llcrypt registered [ 1088.086910] Key type .llcrypt registered [ 1088.212434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1088.222818] alg: No test for adler32 (adler32-zlib) [ 1089.260425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1097.251366] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:38:26 (1789565906) [ 1099.467730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1101.551712] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:38:31 (1789565911) [ 1101.673284] LNet: 59505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.683031] LNetError: 59505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.715054] LNet: Removed LNI 10.1.2.3@tcp [ 1102.641310] Key type .llcrypt unregistered [ 1102.647777] Key type ._llcrypt unregistered [ 1110.139481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1110.505462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1110.515934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1110.533495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1113.394369] Key type ._llcrypt registered [ 1113.396022] Key type .llcrypt registered [ 1113.871388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.962204] alg: No test for adler32 (adler32-zlib) [ 1115.059229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.135985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1126.244830] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:38:56 (1789565936) [ 1126.676546] LNet: 60743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.683810] LNetError: 60743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1126.694605] LNet: Removed LNI 10.1.2.3@tcp [ 1127.602177] Key type .llcrypt unregistered [ 1127.608412] Key type ._llcrypt unregistered [ 1134.606600] Key type ._llcrypt registered [ 1134.611588] Key type .llcrypt registered [ 1134.948385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1134.980951] alg: No test for adler32 (adler32-zlib) [ 1135.950122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.014956] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1146.049270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1147.134085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1147.220558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1147.277650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1147.312653] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1147.649399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.700540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.738829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.775299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.822673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1147.852128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.887717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.933280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.967673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.980727] LNet: 61180:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.015549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.077544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.118325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.174947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.220814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.271649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.350690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.083369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1149.148499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1149.228849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1149.285104] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1149.344128] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1149.679412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.705594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.737669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.775740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.806158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1149.832595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.857979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.884828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.912058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.931956] LNet: 61291:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.956559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.015512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.064289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.153936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.223156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.299976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.364707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.315654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1151.400274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1151.491475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1151.528900] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1151.948178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.987086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.038724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.111829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.163574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1152.222110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.290712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.395331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.509561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.551382] LNet: 61403:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.597911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.668531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.719672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.776574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.841029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.894610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.935284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1153.687924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1153.777093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1153.823425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1153.859459] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1153.874682] LNet: Skipped 1 previous similar message [ 1154.166847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.210454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.268274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.322710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.383872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1154.451045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.508650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.572703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.657171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.699843] LNet: 61514:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.721283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.778797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.833942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.884476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.950921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.012766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.060909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.820608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1155.867567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1155.934689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1156.261351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.316686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.353992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.402740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.453877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1156.505038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.570796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.638776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.674906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.722803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.782054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.855282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.922172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.003693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.075793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.133843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.746276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1157.799434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1157.860285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1157.884415] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1157.891750] LNet: Skipped 1 previous similar message [ 1158.216477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.247565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.287363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.350933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.408738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1158.469519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.535651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.605944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.667293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.737363] LNet: 61737:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.744431] LNet: 61737:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1158.781944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.838320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.883712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.944448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.009510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.073399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.135206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.829452] LNet: 61778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.848417] LNetError: 61778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.880071] LNet: Removed LNI 192.168.203.56@tcp [ 1160.908455] Key type .llcrypt unregistered [ 1160.918321] Key type ._llcrypt unregistered [ 1183.416667] Key type ._llcrypt registered [ 1183.418524] Key type .llcrypt registered [ 1184.944753] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:39:55 (1789565995) [ 1186.063321] Key type .llcrypt unregistered [ 1186.067420] Key type ._llcrypt unregistered [ 1192.281427] Key type ._llcrypt registered [ 1192.284313] Key type .llcrypt registered [ 1192.452765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.475638] alg: No test for adler32 (adler32-zlib) [ 1193.442262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1193.488885] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1201.168726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1213.587426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.706943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1213.738567] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1216.711347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1216.776579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1216.849707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1216.935952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1217.025026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1217.103921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1217.201659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1217.277308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1217.380081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1217.429780] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1217.444528] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1217.822307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.870304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.926541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.011303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.089337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1218.168936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.239201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.283426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.344627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.377862] LNet: 63242:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.422618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.467713] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1218.475866] LNet: Skipped 3 previous similar messages [ 1218.486814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.525941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.587589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.649676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.740370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.837177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.815519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1219.904336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1220.072725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1220.155354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1220.233251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1220.303430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1220.343333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1220.395419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1220.448316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1220.490840] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1220.826630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.894535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.963161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.020990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.067608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1221.120737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.196794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.267539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.329936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.354321] LNet: 63366:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.391240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.461554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.535713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.608735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.704856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.830645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.888206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1222.714129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1222.775881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1222.829724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1222.893625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1222.970710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1223.067439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1223.121390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1223.156144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1223.231040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1223.263179] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1223.279622] LNet: Skipped 2 previous similar messages [ 1223.585594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.632486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.721818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.784587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.872877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1223.945624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.017526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.083101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.108317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.129912] LNet: 63489:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.161977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.225430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.279635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.325439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.372129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.418977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.442799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.844201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1224.884799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1224.925765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1225.004745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1225.068309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1225.117497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1225.161337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1225.202354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1225.249586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1225.808527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.899514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.999525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.112885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.158952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1226.207500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.260978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.298898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.338493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.353848] LNet: 63613:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.392450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.425939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.451608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.513689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.575745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.640076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.777953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.353971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1227.383719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1227.413115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1227.454916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1227.512301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1227.815874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1227.863488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1227.926351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1227.958308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1227.977180] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1227.981185] LNet: Skipped 6 previous similar messages [ 1228.182475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.217772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.275436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.306404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.337608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1228.366822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.404301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.434959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.479909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.509475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.539665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.589786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.613961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.639957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.665519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.690842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.122862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1229.163894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1229.228446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1229.267875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1229.297783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1229.336218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1229.365440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1229.413485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1229.464319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1229.870870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.899825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.943468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.006574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.073996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1230.098192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.124385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.154250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.185427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.219866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.245854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.273513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.299222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.326806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.392811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.463977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.836947] LNet: 63935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.846287] LNetError: 63935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.851624] LNet: Removed LNI 192.168.203.56@tcp [ 1237.867868] LNet: Removed LNI 192.168.203.56@tcp1 [ 1238.548288] Key type .llcrypt unregistered [ 1238.558070] Key type ._llcrypt unregistered [ 1256.602620] Key type ._llcrypt registered [ 1256.607711] Key type .llcrypt registered [ 1258.052685] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:41:08 (1789566068) [ 1258.777397] Key type .llcrypt unregistered [ 1258.781060] Key type ._llcrypt unregistered [ 1265.589499] Key type ._llcrypt registered [ 1265.591951] Key type .llcrypt registered [ 1265.910679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.962450] alg: No test for adler32 (adler32-zlib) [ 1266.847792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.876775] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1275.832660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1276.511649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1276.540231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1276.586844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1276.605326] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1276.916250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.965840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.010913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.059365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.136164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1277.197842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.337595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.400800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.471763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.514407] LNet: 65182:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.541424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.598388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.648636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.692392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.729474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.797023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.851057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.683781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1278.758576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1278.808785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1278.854878] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1279.139710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.177428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.216922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.271894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.308893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1279.355897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.413784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.524773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.588179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.639624] LNet: 65293:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.688833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.739937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.783875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.858933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.947504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.045811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.130609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.768855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1280.812501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1280.866878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1280.881145] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1281.108638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.148476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.184442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.225386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.260542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1281.335515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.371722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.409730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.481481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.505826] LNet: 65404:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.523514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.553506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.581792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.613764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.643059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.670438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.709804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.017229] LNet: 65445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.022989] LNetError: 65445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.039974] LNet: Removed LNI 192.168.203.56@tcp [ 1282.504494] Key type .llcrypt unregistered [ 1282.506351] Key type ._llcrypt unregistered [ 1302.689303] Key type ._llcrypt registered [ 1302.692685] Key type .llcrypt registered [ 1304.315882] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:41:54 (1789566114) [ 1305.321349] Key type .llcrypt unregistered [ 1305.323932] Key type ._llcrypt unregistered [ 1313.105547] Key type ._llcrypt registered [ 1313.113239] Key type .llcrypt registered [ 1313.440468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1313.471797] alg: No test for adler32 (adler32-zlib) [ 1314.381258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1314.413562] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1322.186656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1332.578850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.639180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1332.672215] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1334.399285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1334.446281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1334.478607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1334.531096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1334.578692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1334.635852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1334.714931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1334.801343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1334.877966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1334.911789] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1334.927488] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1335.228741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.254337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.282443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.326190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.369445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1335.395553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.419498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.444158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.470453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.484252] LNet: 66926:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.511160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.542786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.571225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.599313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.634453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.668563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.713108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1335.767578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.377800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1336.423941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1336.485215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1336.539400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1336.606713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1336.668945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1336.718559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1336.760742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1336.811981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1336.829295] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1336.835972] LNet: Skipped 2 previous similar messages [ 1337.037922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1337.065928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1337.091518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1337.113684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1337.142206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1337.187193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1337.234994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1337.279503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1337.339690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1337.361472] LNet: 67052:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1337.378857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.423610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1337.449833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.495351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.542380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.581919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.623468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1337.664290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.177623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1338.236278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1338.283619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1338.331713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1338.361992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1338.396941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1338.429978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1338.490187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1338.557324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1338.580723] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1338.816472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.847278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.891590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.921769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.964932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1339.026528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.068333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.121328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.165909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.186477] LNet: 67177:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.225141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.277187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.321725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.359791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.388914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.426158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.463235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.505531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.096523] LNet: 67255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.119138] LNetError: 67255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.132784] LNet: Removed LNI 192.168.203.56@tcp [ 1345.160749] LNet: Removed LNI 192.168.203.56@tcp1 [ 1345.799662] Key type .llcrypt unregistered [ 1345.806323] Key type ._llcrypt unregistered [ 1364.049846] Key type ._llcrypt registered [ 1364.055395] Key type .llcrypt registered [ 1366.021725] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:42:55 (1789566175) [ 1367.582700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1369.887720] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:43:00 (1789566180) [ 1370.903429] Key type .llcrypt unregistered [ 1370.905174] Key type ._llcrypt unregistered [ 1379.018486] Key type ._llcrypt registered [ 1379.021078] Key type .llcrypt registered [ 1379.250422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1379.278146] alg: No test for adler32 (adler32-zlib) [ 1380.213935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1380.233427] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1388.723617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1389.601560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1389.646593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1389.710850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1389.753745] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1389.996507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1390.034954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1390.082693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1390.132798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1390.188162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1390.244163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1390.282750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1390.308865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1390.350699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1390.374387] LNet: 68797:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1390.402662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1390.430293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1390.454184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1390.477924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1390.496547] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1390.510389] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1390.524546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1390.591564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1391.053499] LNet: 68836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.064548] LNetError: 68836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.100269] LNet: Removed LNI 192.168.203.56@tcp [ 1391.530417] Key type .llcrypt unregistered [ 1391.537585] Key type ._llcrypt unregistered [ 1412.147670] Key type ._llcrypt registered [ 1412.157703] Key type .llcrypt registered [ 1412.392179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.415678] alg: No test for adler32 (adler32-zlib) [ 1413.395569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1413.434719] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1422.641539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1433.703390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.780526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1433.842744] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1436.102863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1436.160936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1436.238352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1436.319214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1436.396763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1436.450716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1436.546555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1436.582313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1436.626766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1436.669219] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1437.030670] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1437.034094] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1437.089862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1437.143307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1437.206887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1437.273843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1437.353057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1437.453858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1437.550929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1437.668347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1437.742847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1437.793603] LNet: 69713:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1437.833597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1437.912384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1438.003952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1438.062522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.158480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1438.260741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.040749] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1444.059444] LNet: Skipped 2 previous similar messages [ 1444.412677] LNet: 69787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.430335] LNetError: 69787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.472898] LNet: Removed LNI 192.168.203.56@tcp [ 1445.475103] LNetError: 69787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.503860] LNet: Removed LNI 192.168.203.56@tcp1 [ 1446.071457] Key type .llcrypt unregistered [ 1446.074838] Key type ._llcrypt unregistered [ 1470.090856] Key type ._llcrypt registered [ 1470.092738] Key type .llcrypt registered [ 1471.987233] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:44:41 (1789566281) [ 1472.479502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.495268] alg: No test for adler32 (adler32-zlib) [ 1473.525440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.623765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.653668] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1474.614976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.645236] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1476.056584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1476.197199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1476.291605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1476.379455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1476.440557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1476.491590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1476.545263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1476.567866] LNetError: 70887:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1476.652054] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1476.655364] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1478.688504] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1479.102160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1482.784459] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1483.867816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1483.915902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1483.985104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.012164] LNet: 70994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.019670] LNetError: 70994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.032817] LNet: Removed LNI 192.168.203.56@tcp [ 1484.045480] LNet: Removed LNI 192.168.203.56@tcp1 [ 1484.113663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.213941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.256095] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1485.218073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.240808] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1486.358310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1486.426556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.559408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1486.656145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1486.711318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1486.785977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1486.833601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1486.887709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1486.926309] LNetError: 71266:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1487.200348] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1490.515370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.344439] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1493.355026] LNet: Skipped 1 previous similar message [ 1498.597322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1498.653098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1498.703515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1506.583750] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:45:17 (1789566317) [ 1506.716475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.760239] LNet: 71975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.793213] LNetError: 71975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.813273] LNetError: 71975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1506.827933] LNet: Removed LNI 192.168.203.56@tcp [ 1506.851200] LNet: Removed LNI 192.168.203.56@tcp1 [ 1506.897417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.993020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.038147] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1507.918623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.945302] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1508.666648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1508.753122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1508.797983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1508.845962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1508.891678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1508.930861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1508.993889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1509.024196] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.56@tcp [ 1509.122745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1510.048737] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1510.053809] LNet: Skipped 1 previous similar message [ 1524.848810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.202746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1525.268842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1525.328130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 500 [ 1525.556470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1532.484561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1532.537591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.552554] LNet: 72475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.561175] LNetError: 72475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.565586] LNetError: 72475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1533.600254] LNet: Removed LNI 192.168.203.56@tcp [ 1533.621099] LNet: Removed LNI 192.168.203.56@tcp1 [ 1533.708684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.808180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.854672] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1534.944730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1534.973220] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1536.033596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1536.061271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1536.113513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1536.158559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1536.225273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1536.282204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1536.317298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1536.377242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1537.889097] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1537.909131] LNet: Skipped 3 previous similar messages [ 1539.046757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.289342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.318499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1548.354907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1556.888139] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:46:07 (1789566367) [ 1557.422758] LNet: 73521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.440885] LNetError: 73521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.451140] LNetError: 73521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1557.465065] LNet: Removed LNI 192.168.203.56@tcp [ 1557.486594] LNet: Removed LNI 192.168.203.56@tcp1 [ 1558.260665] Key type .llcrypt unregistered [ 1558.263552] Key type ._llcrypt unregistered [ 1566.484710] Key type ._llcrypt registered [ 1566.491841] Key type .llcrypt registered [ 1566.729067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.771788] alg: No test for adler32 (adler32-zlib) [ 1567.739796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1567.759543] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1578.331800] Lustre: DEBUG MARKER: Initial discovery [ 1578.518446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1581.299916] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1581.413312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1581.489165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1581.540856] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1581.589633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1581.646731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -l 3 [ 1581.710620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -e local_error [ 1581.752821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m PUT -l 6 [ 1582.113322] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1583.242080] Lustre: DEBUG MARKER: Force oleg356-client.virtnet to discover 192.168.203.156@tcp (in background) [ 1583.357420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1584.990609] Lustre: DEBUG MARKER: Force oleg356-server to discover 192.168.203.56@tcp [ 1586.469072] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1588.126723] Lustre: DEBUG MARKER: Wait for 74272 [ 1590.770379] Lustre: DEBUG MARKER: Finished wait on 74272 [ 1590.850265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1590.932838] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1590.953533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1591.517755] LNet: 74731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.540231] LNetError: 74731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.571197] LNet: Removed LNI 192.168.203.56@tcp [ 1592.280853] Key type .llcrypt unregistered [ 1592.290292] Key type ._llcrypt unregistered [ 1612.515642] Key type ._llcrypt registered [ 1612.518263] Key type .llcrypt registered [ 1614.269332] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:47:04 (1789566424) [ 1614.752476] Key type .llcrypt unregistered [ 1614.760539] Key type ._llcrypt unregistered [ 1621.538283] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1621.736876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1621.745560] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1621.760482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1624.496311] Key type ._llcrypt registered [ 1624.499970] Key type .llcrypt registered [ 1624.766592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1624.788203] alg: No test for adler32 (adler32-zlib) [ 1625.718496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.769214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.789877] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1626.672124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1626.705464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1636.166327] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:47:26 (1789566446) [ 1636.366326] LNet: 76799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.375473] LNetError: 76799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1636.391588] LNet: Removed LNI 10.1.2.3@tcp [ 1636.427888] LNet: Removed LNI 192.168.203.56@tcp [ 1637.281367] Key type .llcrypt unregistered [ 1637.284489] Key type ._llcrypt unregistered [ 1645.761221] Key type ._llcrypt registered [ 1645.770241] Key type .llcrypt registered [ 1646.030563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1646.054229] alg: No test for adler32 (adler32-zlib) [ 1647.084694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.164732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.209410] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1648.974811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1649.030931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1650.420679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.56@tcp [ 1651.495276] LNetError: Unexpected error -22 connecting to 192.168.203.56@tcp at host 192.168.203.56:988 [ 1658.333488] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:47:48 (1789566468) [ 1658.775143] LNet: 77930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.786534] LNetError: 77930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1658.802498] LNet: Removed LNI 10.1.2.3@tcp [ 1658.818692] LNet: Removed LNI 192.168.203.56@tcp [ 1660.000837] Key type .llcrypt unregistered [ 1660.005847] Key type ._llcrypt unregistered [ 1668.912212] Key type ._llcrypt registered [ 1668.914907] Key type .llcrypt registered [ 1669.086792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1669.110214] alg: No test for adler32 (adler32-zlib) [ 1670.054312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.154845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1670.213350] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1671.039772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1671.101301] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1672.544718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.56@tcp1 --nid 192.168.203.56@tcp2 [ 1678.426372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp2 [ 1678.474480] LNetError: 78763:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.56@tcp1 to 192.168.203.56@tcp2 - Route table may be misconfigured [ 1678.511890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp2 192.168.203.56@tcp1 [ 1688.242237] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:48:18 (1789566498) [ 1688.436857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.502906] LNet: 79331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.524949] LNetError: 79331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.540053] LNet: Removed LNI 192.168.203.56@tcp1 [ 1689.577964] LNet: Removed LNI 192.168.203.56@tcp2 [ 1689.673469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.804032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.835474] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1691.133613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.184183] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1692.630535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1692.753491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.56@tcp [ 1692.848343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp1 [ 1692.961572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp -e network_timeout [ 1693.030697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -e network_timeout [ 1693.134993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -e network_timeout [ 1693.237664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -e network_timeout [ 1693.387675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1693.469133] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1693.529879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1693.687254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1693.867233] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1693.899718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1702.511062] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:48:32 (1789566512) [ 1702.656507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.700409] LNet: 80192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.708690] LNetError: 80192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.721449] LNetError: 80192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1702.742407] LNet: Removed LNI 192.168.203.56@tcp [ 1703.784733] LNet: Removed LNI 192.168.203.56@tcp1 [ 1703.898815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.113359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1704.493563] LNet: 80236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.099802] Key type .llcrypt unregistered [ 1705.101264] Key type ._llcrypt unregistered [ 1720.328264] Key type ._llcrypt registered [ 1720.331147] Key type .llcrypt registered [ 1722.143773] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:48:52 (1789566532) [ 1722.719694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.763194] alg: No test for adler32 (adler32-zlib) [ 1723.820708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.259820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1727.344891] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:48:57 (1789566537) [ 1727.502301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.568850] LNet: 81329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.644949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.742066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.800746] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1729.035427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1729.091320] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1730.636157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1730.674635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp1 [ 1730.701473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp1 [ 1738.427930] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:49:09 (1789566549) [ 1740.754776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg356-server [ 1742.537892] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:49:13 (1789566553) [ 1745.108356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg356-server [ 1747.579518] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:49:17 (1789566557) [ 1749.826206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg356-server [ 1752.348522] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:49:22 (1789566562) [ 1754.649710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg356-server [ 1756.930325] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:49:27 (1789566567) [ 1759.383532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg356-server [ 1761.404973] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:49:31 (1789566571) [ 1763.469448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg356-server [ 1767.047726] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:49:36 (1789566576) [ 1770.411040] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg356-server [ 1772.930874] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:49:42 (1789566582) [ 1774.935529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg356-server [ 1776.705280] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:49:47 (1789566587) [ 1779.151957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg356-server [ 1781.139481] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:49:51 (1789566591) [ 1781.408233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.438723] LNet: 85212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.445895] LNetError: 85212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.458914] LNet: Removed LNI 192.168.203.56@tcp [ 1782.503377] LNet: Removed LNI 192.168.203.56@tcp1 [ 1782.575449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.655602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.733410] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1783.709270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1783.797168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1784.059602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.132988] LNet: 85360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.147989] LNetError: 85360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.175101] LNetError: 85360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1784.225539] LNet: Removed LNI 192.168.203.56@tcp [ 1784.293297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.475506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1784.592198] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1785.713279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1787.404592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.463398] LNet: 85514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.478457] LNetError: 85514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.520988] LNet: Removed LNI 192.168.203.56@tcp [ 1787.592244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.783215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.861842] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1789.600947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1789.783622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1790.061935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.107701] LNetError: 85661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.137930] LNet: Removed LNI 192.168.203.56@tcp [ 1790.184836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.270967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1790.331880] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1791.650609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1792.059427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.115087] LNet: 85807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.131517] LNet: 85807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1793.192748] LNet: Removed LNI 192.168.203.56@tcp [ 1793.234297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.294260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.322313] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1794.239893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1794.441739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1794.951491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.977934] LNetError: 85954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.985183] LNetError: 85954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1795.005279] LNet: Removed LNI 192.168.203.56@tcp [ 1795.069716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.248691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1795.311430] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1796.898721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1797.300416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.374708] LNet: Removed LNI 192.168.203.56@tcp [ 1798.425700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.612766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.667777] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1799.967564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1800.109730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1800.496911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.560111] LNet: 86247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.577156] LNet: 86247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1801.642502] LNet: Removed LNI 192.168.203.56@tcp [ 1801.717534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.792572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1801.846546] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1803.173640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1804.873102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.905569] LNetError: 86402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.922806] LNetError: 86402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1804.947660] LNet: Removed LNI 192.168.203.56@tcp [ 1805.002117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.105662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.141633] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1806.512878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1806.638549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1807.141910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.236344] LNet: Removed LNI 192.168.203.56@tcp [ 1808.302879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.448572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1808.501759] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1809.543730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1810.043065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.115235] LNet: Removed LNI 192.168.203.56@tcp [ 1811.201435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.359716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.453709] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1812.572757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1812.712738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1813.107706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.194827] LNet: Removed LNI 192.168.203.56@tcp [ 1814.245756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.369077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1814.455772] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1816.315195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1816.751597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.775401] LNet: 86987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.781714] LNet: 86987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1817.837071] LNet: Removed LNI 192.168.203.56@tcp [ 1817.883322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.921336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1817.956651] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1819.240971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1819.301295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.331948] LNet: Removed LNI 192.168.203.56@tcp [ 1819.353288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.446352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.467471] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1820.649583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1820.758523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.815223] LNet: Removed LNI 192.168.203.56@tcp [ 1820.839915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.913807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1821.100792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.147628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.222849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1821.295218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.376666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.460327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1821.584444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.626826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.739613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1821.802932] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1823.113131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.176965] LNetError: 87427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.190456] LNetError: 87427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1823.243391] LNet: Removed LNI 192.168.203.56@tcp [ 1823.268645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.457282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1833.898859] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:50:43 (1789566643) [ 1834.135557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.248955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.343962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1834.417351] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1836.463779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.514059] LNet: Removed LNI 192.168.203.56@tcp231 [ 1836.593098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.741278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1836.839151] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1836.859099] LNet: 88155:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.060218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1838.149142] LNet: Removed LNI 192.168.203.56@tcp231 [ 1838.188371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1838.252561] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1840.743474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.830299] LNet: Removed LNI 192.168.203.56@tcp231 [ 1840.893617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.089821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1841.198707] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1841.231710] LNet: 88308:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1842.516985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1842.592340] LNet: Removed LNI 192.168.203.56@tcp231 [ 1842.633277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1842.677925] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1845.389520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.446680] LNet: Removed LNI 192.168.203.56@tcp231 [ 1845.477717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.601950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1845.653494] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1845.663502] LNet: 88460:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1846.228831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1846.259785] LNet: Removed LNI 192.168.203.56@tcp231 [ 1846.291159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1846.358346] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1847.886649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.918055] LNet: Removed LNI 192.168.203.56@tcp231 [ 1847.947974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.031370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1848.084478] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1848.094794] LNet: 88611:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1848.942479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1848.993733] LNet: Removed LNI 192.168.203.56@tcp231 [ 1849.020220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1849.055714] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1859.062421] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:51:09 (1789566669) [ 1859.292475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.318701] LNet: 89322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.324217] LNet: 89322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1859.330498] LNetError: 89322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.335795] LNetError: 89322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1859.348493] LNet: Removed LNI 192.168.203.56@tcp231 [ 1859.377806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.447608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1859.493605] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1860.389093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1860.461714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1860.507090] LNet: Removed LNI 192.168.203.56@tcp232 [ 1860.527919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1860.581573] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1861.603618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1861.674819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1861.725981] LNet: Removed LNI 192.168.203.56@tcp232 [ 1861.758264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1861.791637] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1862.596269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1862.630594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1862.656791] LNet: Removed LNI 192.168.203.56@tcp232 [ 1862.668612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1862.692824] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1863.440777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1863.480360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1863.512297] LNet: Removed LNI 192.168.203.56@tcp232 [ 1863.527714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1870.835033] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:51:21 (1789566681) [ 1872.841932] Key type .llcrypt unregistered [ 1872.848492] Key type ._llcrypt unregistered [ 1881.996472] Key type ._llcrypt registered [ 1882.003450] Key type .llcrypt registered [ 1882.436881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.483822] alg: No test for adler32 (adler32-zlib) [ 1883.483170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1883.518563] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1892.960699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1905.598071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.659839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1905.696745] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1907.771947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1907.843849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1907.888714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1907.941818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1907.982966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1908.018207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1908.057448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1908.094928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1908.137055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1908.163844] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1908.170992] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1908.490609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1908.536282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1908.594965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1908.680617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1908.765307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1908.829027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1908.869816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1908.920800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1908.966888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1908.990255] LNet: 91114:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1909.009521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1909.070182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1909.124983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1909.178885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1909.219252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1909.296327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1909.932899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1909.972313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1910.025381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1910.066772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1910.112573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1910.155357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1910.213687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1910.275437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1910.321493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1910.355300] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1910.729395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1910.776837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1910.809901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1910.842166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1910.902470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1910.949586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1911.007457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1911.067520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1911.114852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1911.137276] LNet: 91236:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1911.174833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1911.221743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1911.280698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1911.357435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1911.449964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1911.507643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1912.174699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1912.230655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1912.300041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1912.344316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1912.434932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1912.507269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1912.549166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1912.599608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1912.649978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1912.680622] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1912.984472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1913.055480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1913.132151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1913.173220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1913.221943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1913.281729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1913.352441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1913.423672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1913.503462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1913.557836] LNet: 91359:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1913.596026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1913.662799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1913.707886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1913.765272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1913.838242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1913.886123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1914.558113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1914.629475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1914.688645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1914.755521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1914.824603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1914.923445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1915.000343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1915.063733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1915.163359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1915.196299] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1915.505702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1915.539254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1915.569832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1915.598192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1915.654254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1915.719815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1915.798250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1915.863799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1915.917836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1915.960901] LNet: 91481:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1915.977731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1916.017434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1916.043696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1916.089755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1916.147302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1916.194033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1916.791708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1916.836541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1916.885402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1916.942530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1916.992715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1917.031954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1917.065803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1917.100724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1917.143276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1917.449875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1917.492446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1917.555054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1917.618782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1917.668645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1917.722722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1917.774990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1917.854722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1917.916322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1917.949288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1918.001630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1918.050161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1918.090712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1918.135372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1918.168369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1918.798184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1918.875887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1918.971666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1919.048811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1919.116101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1919.197785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1919.302680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1919.406892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1919.479099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1919.519593] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1919.533101] LNet: Skipped 1 previous similar message [ 1919.555337] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1919.935813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1919.991984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1920.039841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1920.096077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1920.151455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1920.225427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1920.302944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1920.334229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1920.407538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1920.446621] LNet: 91725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.459723] LNet: 91725:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1920.488625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1920.548248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1920.596234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1920.630914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1920.668887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1920.715109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1926.485228] LNet: 91800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.491494] LNetError: 91800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.501869] LNet: Removed LNI 192.168.203.56@tcp [ 1926.517791] LNet: Removed LNI 192.168.203.56@tcp1 [ 1926.939888] Key type .llcrypt unregistered [ 1926.943746] Key type ._llcrypt unregistered [ 1948.031225] Key type ._llcrypt registered [ 1948.036346] Key type .llcrypt registered [ 1949.532754] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:52:39 (1789566759) [ 1950.014676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.059037] alg: No test for adler32 (adler32-zlib) [ 1951.321413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.487266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1951.590728] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1953.303910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1953.359453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1953.398438] LNetError: 92776:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1953.440619] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1953.573161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.56@tcp [ 1953.777989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1961.843201] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:52:52 (1789566772) [ 1962.058494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.116405] LNet: 93364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.130129] LNetError: 93364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.153756] LNet: Removed LNI 192.168.203.56@tcp [ 1962.191260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.274375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.304370] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1963.316430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1963.341888] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1964.822047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.56@tcp [ 1964.865536] LNetError: 93632:0:(api-ni.c:4493:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1965.346727] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1966.259575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.56@tcp [ 1974.936828] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:53:05 (1789566785) [ 1977.410941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg356-server [ 1979.479920] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:53:09 (1789566789) [ 1981.510746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg356-server [ 1983.610248] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:53:14 (1789566794) [ 1986.304345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg356-server [ 1988.851248] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:53:18 (1789566798) [ 1988.998808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.054464] LNet: 95250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.074761] LNetError: 95250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.092179] LNet: Removed LNI 192.168.203.56@tcp [ 1989.123134] LNet: Removed LNI 192.168.203.56@tcp1 [ 1989.194266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.272339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.305309] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1990.106772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1990.200682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1990.253396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1990.370719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1990.417054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1990.574903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1998.458870] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:53:28 (1789566808) [ 1998.670677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.716095] LNet: 95982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.734257] LNetError: 95982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.763471] LNetError: 95982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1998.830207] LNet: Removed LNI 192.168.203.56@tcp [ 1998.886493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.058170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1999.121498] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2006.854806] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:53:37 (1789566817) [ 2007.107704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.146239] LNet: 96690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.154658] LNetError: 96690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.171597] LNet: Removed LNI 192.168.203.56@tcp [ 2007.190242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.272232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.306175] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2015.600770] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:53:46 (1789566826) [ 2017.314456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2019.430499] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:53:50 (1789566830) [ 2019.913195] LNet: 97726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.926542] LNetError: 97726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.959628] LNet: Removed LNI 192.168.203.56@tcp [ 2020.650603] Key type .llcrypt unregistered [ 2020.654520] Key type ._llcrypt unregistered [ 2028.989468] Key type ._llcrypt registered [ 2028.991462] Key type .llcrypt registered [ 2029.429243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2029.499886] alg: No test for adler32 (adler32-zlib) [ 2030.470749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2030.506412] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2040.035535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2046.646736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.156@tcp [ 2047.125154] LNet: 98142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.137655] LNetError: 98142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.155821] LNet: Removed LNI 192.168.203.56@tcp [ 2047.802975] Key type .llcrypt unregistered [ 2047.811468] Key type ._llcrypt unregistered [ 2062.745249] Key type ._llcrypt registered [ 2062.747348] Key type .llcrypt registered [ 2064.327983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2066.627415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2068.775512] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:54:39 (1789566879) [ 2070.540982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2073.024845] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:54:43 (1789566883) [ 2075.421251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg356-server [ 2077.624392] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:54:48 (1789566888) [ 2079.804667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg356-server [ 2082.020324] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:54:52 (1789566892) [ 2083.314060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.423196] alg: No test for adler32 (adler32-zlib) [ 2084.642936] Lustre: Lustre: Build Version: 2.17.58_39_gf62a9ec [ 2085.003860] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2086.744275] Key type lgssc registered [ 2088.315390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2112.413745] Key type lgssc unregistered [ 2112.721261] LNet: 101316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.736647] LNetError: 101316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.771311] LNet: Removed LNI 192.168.203.56@tcp [ 2113.817749] Key type .llcrypt unregistered [ 2113.822139] Key type ._llcrypt unregistered [ 2133.824317] Key type ._llcrypt registered [ 2133.827112] Key type .llcrypt registered [ 2135.429386] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:55:45 (1789566945) [ 2135.805073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.820554] alg: No test for adler32 (adler32-zlib) [ 2136.805675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.911806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2136.955314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2137.011543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2137.073504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2144.838885] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:55:55 (1789566955) [ 2145.191138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.238585] LNet: 102729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.288842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.392402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.437034] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2146.378829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.398855] LNet: 102873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.406941] LNetError: 102873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.426700] LNet: Removed LNI 192.168.203.56@tcp [ 2146.457980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.557933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2146.606642] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2147.304896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.352840] LNetError: 102902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.377795] LNet: Removed LNI 192.168.203.56@tcp [ 2147.423204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.473252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2147.492748] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2155.766232] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:56:05 (1789566965) [ 2156.119776] LNet: 103492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.124991] LNet: 103492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2156.133618] LNetError: 103492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.145141] LNet: Removed LNI 192.168.203.56@tcp [ 2157.160079] Key type .llcrypt unregistered [ 2157.164823] Key type ._llcrypt unregistered [ 2167.066985] Key type ._llcrypt registered [ 2167.070718] Key type .llcrypt registered [ 2167.388244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.411666] alg: No test for adler32 (adler32-zlib) [ 2168.325417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.399083] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2168.402902] LNet: *** cfs_fail_loc=e003, val=100*** [ 2169.168189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2169.191189] LNet: *** cfs_fail_loc=e003, val=100*** [ 2169.194837] LNet: Skipped 2 previous similar messages [ 2169.200163] Ping reply received with mapping: [ 2169.201951] found metadata reply NI[192.168.203.56@tcp] [ 2169.209220] 38 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2169.216132] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2169.220374] 5a 5a 5a 5a 5a 5a 5a 5a [ 2169.539294] LNet: 103888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.550932] LNetError: 103888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.593171] LNet: Removed LNI 192.168.203.56@tcp [ 2170.535098] Key type .llcrypt unregistered [ 2170.539276] Key type ._llcrypt unregistered [ 2186.201847] Key type ._llcrypt registered [ 2186.204241] Key type .llcrypt registered [ 2188.172283] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:56:38 (1789566998) [ 2189.050848] Key type .llcrypt unregistered [ 2189.053724] Key type ._llcrypt unregistered [ 2198.267342] Key type ._llcrypt registered [ 2198.271294] Key type .llcrypt registered [ 2198.447467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2198.479559] alg: No test for adler32 (adler32-zlib) [ 2199.624875] LNetError: 104927:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2200.041389] Key type .llcrypt unregistered [ 2200.047079] Key type ._llcrypt unregistered [ 2206.283047] Key type ._llcrypt registered [ 2206.287282] Key type .llcrypt registered [ 2207.766321] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:56:58 (1789567018) [ 2208.653271] Key type .llcrypt unregistered [ 2208.658271] Key type ._llcrypt unregistered [ 2217.162778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2217.171399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2218.830520] Key type ._llcrypt registered [ 2218.833655] Key type .llcrypt registered [ 2219.193163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2219.233840] alg: No test for adler32 (adler32-zlib) [ 2220.176930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.223155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2232.029613] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:57:22 (1789567042) [ 2232.345789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.397360] LNet: 106694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.408321] LNetError: 106694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2232.435045] LNet: Removed LNI 10.1.2.3@tcp [ 2232.461875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.428661] Key type .llcrypt unregistered [ 2233.430992] Key type ._llcrypt unregistered [ 2240.916870] Key type ._llcrypt registered [ 2240.921944] Key type .llcrypt registered [ 2241.379373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2241.419953] alg: No test for adler32 (adler32-zlib) [ 2242.426298] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2242.551470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2242.805312] LNet: 106980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.812690] LNetError: 106980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.832932] LNet: Removed LNI 192.168.203.56@tcp [ 2243.880496] Key type .llcrypt unregistered [ 2243.884492] Key type ._llcrypt unregistered [ 2258.282919] Key type ._llcrypt registered [ 2258.285338] Key type .llcrypt registered [ 2259.974271] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:57:50 (1789567070) [ 2260.896433] Key type .llcrypt unregistered [ 2260.900506] Key type ._llcrypt unregistered [ 2268.766316] Key type ._llcrypt registered [ 2268.774293] Key type .llcrypt registered [ 2269.114777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2269.161195] alg: No test for adler32 (adler32-zlib) [ 2278.252629] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:58:08 (1789567088) [ 2278.412702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.469878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.570794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2278.595944] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2279.290467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.333418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2279.372275] LNet: Removed LNI 192.168.203.56@tcp [ 2279.742544] LNet: 108724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.206405] Key type .llcrypt unregistered [ 2280.210963] Key type ._llcrypt unregistered [ 2293.554422] Key type ._llcrypt registered [ 2293.559960] Key type .llcrypt registered [ 2295.061590] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:58:25 (1789567105) [ 2295.619508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2295.641072] alg: No test for adler32 (adler32-zlib) [ 2296.686926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.779512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.808994] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2297.846087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2297.923278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.56@tcp [ 2297.953748] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2297.960932] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2297.967783] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2297, 0, 0 [ 2297.979557] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2297.984606] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2297.987908] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2297.996313] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2297, 0, 0 [ 2298.004949] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2298.013960] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2298.018971] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2298.022621] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2297, 0, 2347 [ 2298.028366] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2298.032581] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2298.037441] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2298.044470] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2297, 0, 2347 [ 2298.053282] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2305.828820] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:58:36 (1789567116) [ 2306.632957] LNet: 110264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.647267] LNetError: 110264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.672538] LNet: Removed LNI 192.168.203.56@tcp [ 2307.236697] Key type .llcrypt unregistered [ 2307.240112] Key type ._llcrypt unregistered [ 2314.723386] Key type ._llcrypt registered [ 2314.726278] Key type .llcrypt registered [ 2314.993741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2315.017685] alg: No test for adler32 (adler32-zlib) [ 2315.993045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2316.074474] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2325.069958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2335.810163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.910446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2335.942853] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 2337.102599] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2337.389674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2337.394125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2337.401933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2339.772994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2339.790708] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2353.194070] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:59:23 (1789567163) [ 2353.452221] LNet: 111764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.474358] LNetError: 111764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.501643] LNet: Removed LNI 192.168.203.56@tcp [ 2354.529405] LNet: Removed LNI 192.168.203.56@tcp1 [ 2354.541195] LNetError: 111764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2354.549790] LNetError: 111764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2354.577387] LNet: Removed LNI 10.1.2.3@tcp99 [ 2355.593475] Key type .llcrypt unregistered [ 2355.596427] Key type ._llcrypt unregistered [ 2363.825304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2363.830296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2366.542136] Key type ._llcrypt registered [ 2366.546475] Key type .llcrypt registered [ 2366.750059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2366.792274] alg: No test for adler32 (adler32-zlib) [ 2367.777561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.838570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.911811] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2368.983506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2369.005763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2371.627490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2371.721498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2371.772311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2371.802562] LNetError: 112508:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2371.853987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2371.904983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2372.375120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2372.483373] LNet: 112529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.494591] LNetError: 112529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2372.506444] LNet: Removed LNI 10.1.2.3@tcp [ 2373.544368] LNet: Removed LNI 192.168.203.56@tcp [ 2374.303651] Key type .llcrypt unregistered [ 2374.307170] Key type ._llcrypt unregistered [ 2389.734383] Key type ._llcrypt registered [ 2389.737048] Key type .llcrypt registered [ 2391.463823] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:00:01 (1789567201) [ 2392.034498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2392.066275] alg: No test for adler32 (adler32-zlib) [ 2393.069272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.155262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2393.199484] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2394.114980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2394.224236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg356-client@tcp [ 2402.098847] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:00:12 (1789567212) [ 2402.675481] LNet: 114060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.695200] LNetError: 114060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2402.736385] LNet: Removed LNI 192.168.203.56@tcp [ 2403.645331] Key type .llcrypt unregistered [ 2403.654654] Key type ._llcrypt unregistered [ 2411.924244] Key type ._llcrypt registered [ 2411.928737] Key type .llcrypt registered [ 2412.457383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.546577] alg: No test for adler32 (adler32-zlib) [ 2413.742128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2413.782594] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2413.850854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2422.605272] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:00:32 (1789567232) [ 2424.486930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2427.335962] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:00:36 (1789567236) [ 2430.480258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2433.623836] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:00:43 (1789567243) [ 2433.785416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.851238] LNet: 115477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.859307] LNetError: 115477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.898899] LNet: Removed LNI 192.168.203.56@tcp [ 2433.945338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2434.107498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2434.163948] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2435.293063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2435.386381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2445.782503] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:00:55 (1789567255) [ 2446.078306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.118023] LNet: 116191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.143752] LNetError: 116191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.177600] LNet: Removed LNI 192.168.203.56@tcp [ 2446.212430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.252413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2446.277509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2446.861315] Key type .llcrypt unregistered [ 2446.864293] Key type ._llcrypt unregistered [ 2460.214417] Key type ._llcrypt registered [ 2460.216411] Key type .llcrypt registered [ 2461.671881] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:01:12 (1789567272) [ 2462.127570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.146119] alg: No test for adler32 (adler32-zlib) [ 2463.104604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.165699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2463.200839] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2464.031882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2464.128292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2471.397317] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:01:22 (1789567282) [ 2471.537862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.588279] LNet: 117755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.603338] LNetError: 117755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.632279] LNet: Removed LNI 192.168.203.56@tcp [ 2471.688674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.759459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2471.789321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2479.413767] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:01:29 (1789567289) [ 2479.558685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.616360] LNet: 118335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.657188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.784419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2480.543249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2480.612611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2480.731196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2481.349678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2481.450994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2481.572581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.622453] LNet: 118374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.662786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.788550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2482.429543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2482.484453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2482.566687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2483.428213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2483.488679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2492.924648] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:01:43 (1789567303) [ 2493.045293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2493.083330] LNet: 118969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.136779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.223360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2493.247870] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2494.312403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2494.336903] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2495.271315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2495.394059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2495.676699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2495.709595] LNetError: 119255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.739582] LNet: Removed LNI 192.168.203.56@tcp [ 2496.750577] LNetError: 119255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.789021] LNet: Removed LNI 192.168.203.56@tcp2 [ 2496.829794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.015260] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2497.027391] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2497.037737] LNet: 119261:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.092727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2498.403287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2498.430128] LNet: 119298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.435684] LNet: 119298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2499.488416] LNet: Removed LNI 192.168.203.56@tcp [ 2499.496154] LNetError: 119298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.520563] LNetError: 119298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2499.548495] LNet: Removed LNI 192.168.203.56@tcp2 [ 2499.587350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2499.701431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2499.747641] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2500.637665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2500.666236] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2501.773498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2501.831425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2501.918164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2502.212494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2503.264144] LNet: Removed LNI 192.168.203.56@tcp [ 2504.299226] LNet: Removed LNI 192.168.203.56@tcp2 [ 2504.364971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.461205] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2504.485701] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2504.493081] LNet: 119589:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2505.071673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2505.276328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2505.323478] LNetError: 119627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.334245] LNetError: 119627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2505.347923] LNet: Removed LNI 192.168.203.56@tcp [ 2506.407854] LNet: Removed LNI 192.168.203.56@tcp2 [ 2506.479927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.606679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2506.658199] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2507.750537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2507.774274] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2508.686476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2508.747992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2508.941899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2508.961836] LNet: 119906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.970283] LNet: 119906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2509.984268] LNet: Removed LNI 192.168.203.56@tcp [ 2510.009930] LNet: Removed LNI 192.168.203.56@tcp2 [ 2510.050640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.149986] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2510.154768] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2510.161684] LNet: 119912:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2510.928985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2511.226772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2512.353219] LNet: Removed LNI 192.168.203.56@tcp [ 2512.370777] LNet: Removed LNI 192.168.203.56@tcp2 [ 2512.442943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.550485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2512.612863] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2513.898670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2513.949293] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2514.847737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2514.917182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2515.008283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2515.258825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2515.288753] LNetError: 120234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.302283] LNetError: 120234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2515.320774] LNet: Removed LNI 192.168.203.56@tcp [ 2516.409338] LNet: Removed LNI 192.168.203.56@tcp2 [ 2516.471769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2516.579303] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2516.596531] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2516.608945] LNet: 120241:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2517.132582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2517.331589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2517.355323] LNet: Removed LNI 192.168.203.56@tcp [ 2518.379948] LNet: Removed LNI 192.168.203.56@tcp2 [ 2518.428303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2518.517592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2518.545270] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2519.332390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2519.366579] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2520.541183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2520.587421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2520.655964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2521.097287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2522.208258] LNet: Removed LNI 192.168.203.56@tcp [ 2522.224797] LNet: Removed LNI 192.168.203.56@tcp2 [ 2522.300747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.396881] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2522.412252] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2522.423732] LNet: 120581:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2523.126236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2523.477119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.576318] LNet: Removed LNI 192.168.203.56@tcp [ 2524.595128] LNet: Removed LNI 192.168.203.56@tcp2 [ 2524.637981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.720220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2524.743596] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2525.791349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2525.835677] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2526.691410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2526.725143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2526.764505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2526.854513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2527.320828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2527.347496] LNet: 120935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.356105] LNet: 120935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2528.416142] LNet: Removed LNI 192.168.203.56@tcp [ 2529.454697] LNet: Removed LNI 192.168.203.56@tcp2 [ 2529.527804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2529.657837] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2529.676324] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2530.786301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2539.671963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2541.501124] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:02:32 (1789567352) [ 2541.781680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2541.815776] LNetError: 121701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2541.824351] LNetError: 121701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2541.835301] LNet: Removed LNI 192.168.203.56@tcp [ 2542.890048] LNet: Removed LNI 192.168.203.56@tcp2 [ 2542.923026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2543.269487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2543.316918] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2544.216740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2544.244222] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2545.212690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2545.269102] LNet: Removed LNI 10.1.2.3@tcp [ 2545.298798] LNet: Removed LNI 192.168.203.56@tcp [ 2545.328848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.424929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2545.464965] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2545.480729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2546.185821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2546.226792] LNet: Removed LNI 192.168.203.56@tcp [ 2546.241587] LNet: Removed LNI 10.1.2.3@tcp [ 2553.749416] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:02:44 (1789567364) [ 2553.846536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2553.921398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2554.026696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2554.058257] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2555.172839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2555.224050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2555.316791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.413442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.488333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.584353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.644452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.691144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.756384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.798100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.827101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2555.867393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2556.050425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2564.980115] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 10:02:55 (1789567375) [ 2565.526652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2565.555129] LNet: 123318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.570132] LNet: 123318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2566.635567] LNet: Removed LNI 192.168.203.56@tcp [ 2566.697448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.850439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2566.910719] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2568.044346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2568.092096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2568.171660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.249215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.313903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.368848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.433873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.488700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.524565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.601412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.687027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.778496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2568.920468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2568.993863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.072361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.134845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.191583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.247810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.311921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.374316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.488196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.607397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2569.667568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2577.459350] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:03:08 (1789567388) [ 2577.620781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2577.655239] LNetError: 124068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.661664] LNetError: 124068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2577.690478] LNet: Removed LNI 192.168.203.56@tcp [ 2577.729051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2577.802220] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2577.947675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2577.957591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2577.966514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2580.456794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2580.538504] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2581.639591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2581.685850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2584.737357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2584.806089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2584.888713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.56@tcp -m PUT -e local_timeout -r 1 [ 2584.979528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2585.073267] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2585.077196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2585.343929] LNetError: 124142:0:(lib-move.c:4794:lnet_parse()) 192.168.203.56@tcp, src 192.168.203.56@tcp: Dropping PUT (error -108 looking up sender) [ 2586.340152] LNet: Removed LNI 10.1.2.3@tcp [ 2587.366207] LNet: Removed LNI 192.168.203.56@tcp [ 2587.905899] Key type .llcrypt unregistered [ 2587.907900] Key type ._llcrypt unregistered [ 2602.548873] Key type ._llcrypt registered [ 2602.550101] Key type .llcrypt registered [ 2604.188133] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:03:34 (1789567414) [ 2604.984322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.016197] alg: No test for adler32 (adler32-zlib) [ 2606.048722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2606.534936] LNet: 125453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.300456] Key type .llcrypt unregistered [ 2607.302497] Key type ._llcrypt unregistered [ 2616.128563] Key type ._llcrypt registered [ 2616.134472] Key type .llcrypt registered [ 2616.438642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2616.473150] alg: No test for adler32 (adler32-zlib) [ 2617.637987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2617.724862] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2627.089649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2627.334460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2628.125682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2628.405106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2628.445985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2628.488551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2628.562726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2628.607347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2628.649558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2628.693679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2628.750600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2628.793555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2628.821132] LNet: 125903:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2628.868467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2628.927400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2628.972863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2629.014807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2629.027827] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2629.064925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2629.153367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2630.174809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2630.562175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2630.616033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2630.651381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2630.721637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2630.762543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2630.803902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2630.837944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2630.881952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2630.941942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2630.962451] LNet: 126022:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2631.003667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2631.088685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2631.145681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2631.212944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2631.309694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2631.366314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2632.279056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2632.644217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2632.695224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2632.747538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2632.805929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2632.882987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2632.924233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2632.989980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2633.059274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2633.152478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2633.189537] LNet: 126141:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2633.220946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2633.287549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2633.354337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2633.437289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2633.526994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2633.628351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2634.622093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2635.031453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2635.112160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2635.172452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2635.210559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2635.267793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2635.353205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2635.415715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2635.459938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2635.529543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2635.572823] LNet: 126261:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2635.604761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2635.653385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2635.692874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2635.786070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2635.870287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2635.959754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2636.481286] LNet: 126307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.505402] LNetError: 126307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2636.525993] LNet: Removed LNI 192.168.203.56@tcp [ 2637.156317] Key type .llcrypt unregistered [ 2637.161567] Key type ._llcrypt unregistered [ 2659.735127] Key type ._llcrypt registered [ 2659.742223] Key type .llcrypt registered [ 2661.439094] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:04:31 (1789567471) [ 2662.138425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.186682] alg: No test for adler32 (adler32-zlib) [ 2663.212632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2663.783425] LNet: 127194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.413682] Key type .llcrypt unregistered [ 2664.418163] Key type ._llcrypt unregistered [ 2672.188285] Key type ._llcrypt registered [ 2672.191473] Key type .llcrypt registered [ 2672.434443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2672.451978] alg: No test for adler32 (adler32-zlib) [ 2673.364589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2673.423722] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2683.323649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2694.456990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2694.576200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2694.644836] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 2695.884786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2696.593663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2696.828788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2696.882559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2696.934366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2696.991885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2696.992146] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2697.068392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2697.107896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2697.142445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2697.215881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2697.266307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2697.301846] LNet: 127848:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2697.335120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2697.380911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2697.428180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2697.465606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2697.527553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2697.581060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2698.265542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2698.567852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2698.621232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2698.654561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2698.704770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2698.760941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2698.799567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2698.840441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2698.897283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2698.942560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2698.969989] LNet: 127967:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2699.017566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2699.065386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2699.120575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2699.190114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2699.284451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2699.331290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2700.310239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2700.632036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2700.672482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2700.705306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2700.738164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2700.767278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2700.799489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2700.837478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2700.877604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2700.908173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2700.954225] LNet: 128087:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2700.985888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2701.042603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2701.104423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2701.153945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2701.210334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2701.291810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2702.033895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2702.374738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2702.438600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2702.500750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2702.551880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2702.625562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2702.663509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2702.696812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2702.726847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2702.756545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2702.791944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2702.861249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2702.900493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2702.963281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2703.029894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2703.082654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2709.331798] LNet: 128287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.337304] LNetError: 128287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2709.349279] LNet: Removed LNI 192.168.203.56@tcp [ 2709.365898] LNet: Removed LNI 192.168.203.56@tcp1 [ 2709.914961] Key type .llcrypt unregistered [ 2709.918962] Key type ._llcrypt unregistered [ 2729.896543] Key type ._llcrypt registered [ 2729.899516] Key type .llcrypt registered [ 2731.312336] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:05:41 (1789567541) [ 2731.952747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.978100] alg: No test for adler32 (adler32-zlib) [ 2732.950351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.018266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2733.037892] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2733.867885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2733.900986] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2735.130628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2735.315861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2736.418163] LNet: Removed LNI 192.168.203.56@tcp [ 2744.121580] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:05:54 (1789567554) [ 2745.828195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2747.817640] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:05:58 (1789567558) [ 2758.099231] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:06:07 (1789567567) [ 2766.337644] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:06:16 (1789567576) [ 2777.228877] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:06:27 (1789567587) [ 2785.632343] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:06:36 (1789567596) [ 2793.862289] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:06:44 (1789567604) [ 2802.376937] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:06:53 (1789567613) [ 2810.877709] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:07:01 (1789567621) [ 2820.076702] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:07:10 (1789567630) [ 2828.375993] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:07:18 (1789567638) [ 2837.140857] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:07:27 (1789567647) [ 2845.456596] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:07:36 (1789567656) [ 2854.421116] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:07:44 (1789567664) [ 2862.776019] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:07:53 (1789567673) [ 2871.451095] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:08:02 (1789567682) [ 2879.868671] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:08:10 (1789567690) [ 2888.947931] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:08:19 (1789567699) [ 2896.789551] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:08:27 (1789567707) [ 2897.071721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2897.108764] LNet: 139991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2897.114912] LNetError: 139991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2897.134428] LNet: Removed LNI 192.168.203.56@tcp2 [ 2897.197779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2897.373865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2897.394403] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2898.841888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2898.888315] LNet: 140133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2898.903385] LNetError: 140133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2898.935050] LNet: Removed LNI 192.168.203.56@tcp [ 2899.002613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2899.090570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2899.124924] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2906.762422] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:08:37 (1789567717) [ 2915.724412] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:08:46 (1789567726) [ 2926.206141] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:08:56 (1789567736) [ 2935.404166] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:09:05 (1789567745) [ 2943.075656] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:09:13 (1789567753) [ 2951.557978] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:09:22 (1789567762) [ 2960.965967] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:09:31 (1789567771) [ 2970.496100] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:09:41 (1789567781) [ 2981.079736] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:09:51 (1789567791) [ 2990.969451] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:10:01 (1789567801) [ 2999.476833] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:10:09 (1789567809) [ 3007.435160] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:10:18 (1789567818) [ 3014.614373] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2789 sec ========== 10:10:25 (1789567825) [ 3015.025509] LNet: 147606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3015.040562] LNetError: 147606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3015.059312] LNet: Removed LNI 192.168.203.56@tcp [ 3015.827676] Key type .llcrypt unregistered [ 3015.831450] Key type ._llcrypt unregistered [ 3031.174176] Key type ._llcrypt registered [ 3031.177771] Key type .llcrypt registered [ 3031.992845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3032.030489] alg: No test for adler32 (adler32-zlib) [ 3033.225507] Lustre: Lustre: Build Version: 2.17.58_39_gf62a9ec [ 3033.666276] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 3035.506456] Key type lgssc registered [ 3037.128552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.369938] Lustre: Mounted lustre-client - version 2.17.58_39_gf62a9ec [ 3119.699121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3129.374626] Lustre: Unmounted lustre-client [ 3151.032462] Key type lgssc unregistered [ 3151.326709] LNet: 150556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3151.349196] LNetError: 150556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.423642] LNet: Removed LNI 192.168.203.56@tcp [ 3153.355368] Key type .llcrypt unregistered [ 3153.361071] Key type ._llcrypt unregistered