[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 421132225 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002263] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006736] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009161] LSM: Security Framework initializing [ 0.010044] Yama: becoming mindful. [ 0.011031] SELinux: Initializing. [ 0.012050] *** VALIDATE selinux *** [ 0.019599] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024387] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027098] *** VALIDATE tmpfs *** [ 0.029373] *** VALIDATE proc *** [ 0.030223] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034054] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.039926] debug: unmapping init [mem 0xffffffff89659000-0xffffffff89660fff] [ 0.041880] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042681] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043017] ... version: 2 [ 0.043886] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.045009] ... value mask: 0000ffffffffffff [ 0.046148] ... max period: 00007fffffffffff [ 0.047011] ... fixed-purpose events: 3 [ 0.047994] ... event mask: 000000070000000f [ 0.049254] rcu: Hierarchical SRCU implementation. [ 0.051569] smp: Bringing up secondary CPUs ... [ 0.052594] x86: Booting SMP configuration: [ 0.053018] .... node #0, CPUs: #1 #2 #3 [ 0.056201] smp: Brought up 1 node, 4 CPUs [ 0.057853] smpboot: Max logical packages: 1 [ 0.058011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234610] node 0 deferred pages initialised in 174ms [ 0.238196] devtmpfs: initialized [ 0.239305] x86/mm: Memory block size: 128MB [ 0.241000] gcov: version magic: 0x41383552 [ 0.244216] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.248071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251286] pinctrl core: initialized pinctrl subsystem [ 0.253152] [ 0.253892] ************************************************************* [ 0.256022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259018] ** ** [ 0.262018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266017] ** ** [ 0.269015] ** This means that this kernel is built to expose internal ** [ 0.272024] ** IOMMU data structures, which may compromise security on ** [ 0.275042] ** your system. ** [ 0.278033] ** ** [ 0.281020] ** If you see this message and you are not debugging the ** [ 0.284015] ** kernel, report this immediately to your vendor! ** [ 0.286010] ** ** [ 0.288012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290007] ************************************************************* [ 0.292687] NET: Registered protocol family 16 [ 0.295426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.297062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.304446] cpuidle: using governor menu [ 0.307474] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.309546] PCI: Using configuration type 1 for base access [ 0.311138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.320023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322047] cryptd: max_cpu_qlen set to 1000 [ 0.324236] ACPI: Added _OSI(Module Device) [ 0.325010] ACPI: Added _OSI(Processor Device) [ 0.326008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.332854] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.342754] ACPI: Interpreter enabled [ 0.344086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.346013] ACPI: Using IOAPIC for interrupt routing [ 0.347132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350381] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.358336] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.360041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.362018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.364094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368000] acpiphp: Slot [2] registered [ 0.369123] acpiphp: Slot [5] registered [ 0.370092] acpiphp: Slot [6] registered [ 0.371093] acpiphp: Slot [3] registered [ 0.372066] acpiphp: Slot [4] registered [ 0.373068] acpiphp: Slot [7] registered [ 0.374077] acpiphp: Slot [8] registered [ 0.375074] acpiphp: Slot [9] registered [ 0.376018] acpiphp: Slot [10] registered [ 0.376811] acpiphp: Slot [11] registered [ 0.378046] acpiphp: Slot [12] registered [ 0.379089] acpiphp: Slot [13] registered [ 0.380084] acpiphp: Slot [14] registered [ 0.381071] acpiphp: Slot [15] registered [ 0.382052] acpiphp: Slot [16] registered [ 0.384107] acpiphp: Slot [17] registered [ 0.385102] acpiphp: Slot [18] registered [ 0.387237] acpiphp: Slot [19] registered [ 0.389264] acpiphp: Slot [20] registered [ 0.391095] acpiphp: Slot [21] registered [ 0.393102] acpiphp: Slot [22] registered [ 0.394086] acpiphp: Slot [23] registered [ 0.396077] acpiphp: Slot [24] registered [ 0.397081] acpiphp: Slot [25] registered [ 0.399104] acpiphp: Slot [26] registered [ 0.400062] acpiphp: Slot [27] registered [ 0.402085] acpiphp: Slot [28] registered [ 0.403064] acpiphp: Slot [29] registered [ 0.404098] acpiphp: Slot [30] registered [ 0.406078] acpiphp: Slot [31] registered [ 0.407218] PCI host bridge to bus 0000:00 [ 0.409013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.416019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421264] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.425139] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427369] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435630] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.438756] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.445013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447621] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.449873] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.451022] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.453795] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.465014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.469011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475489] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.486016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510508] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.514015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.519037] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.531015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.545126] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.547352] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.550333] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.552329] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.554249] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.560112] iommu: Default domain type: Passthrough [ 0.561424] SCSI subsystem initialized [ 0.563096] ACPI: bus type USB registered [ 0.564110] usbcore: registered new interface driver usbfs [ 0.566037] usbcore: registered new interface driver hub [ 0.567060] usbcore: registered new device driver usb [ 0.569158] pps_core: LinuxPPS API ver. 1 registered [ 0.570009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.572041] PTP clock support registered [ 0.573127] EDAC MC: Ver: 3.0.0 [ 0.575101] PCI: Using ACPI for IRQ routing [ 0.576000] NetLabel: Initializing [ 0.577008] NetLabel: domain hash size = 128 [ 0.579009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.580073] NetLabel: unlabeled traffic allowed by default [ 0.582109] vgaarb: loaded [ 0.583240] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.585013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.591000] clocksource: Switched to clocksource kvm-clock [ 0.694696] VFS: Disk quotas dquot_6.6.0 [ 0.695890] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.698285] *** VALIDATE ramfs *** [ 0.699310] *** VALIDATE hugetlbfs *** [ 0.700614] pnp: PnP ACPI init [ 0.702779] pnp: PnP ACPI: found 6 devices [ 0.718501] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.721224] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.723016] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.724470] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.726483] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.728404] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.730754] NET: Registered protocol family 2 [ 0.732804] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.737773] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.740814] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.745954] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.748844] TCP: Hash tables configured (established 65536 bind 65536) [ 0.750862] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.752887] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.755027] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.757403] NET: Registered protocol family 1 [ 0.759877] RPC: Registered named UNIX socket transport module. [ 0.761754] RPC: Registered udp transport module. [ 0.762952] RPC: Registered tcp transport module. [ 0.764606] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.769084] NET: Registered protocol family 44 [ 0.770820] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.772764] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.773841] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.776364] PCI: CLS 0 bytes, default 64 [ 0.778523] Unpacking initramfs... [ 2.061384] debug: unmapping init [mem 0xffff95c17cc64000-0xffff95c17ffcffff] [ 2.064522] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.066280] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.069103] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.516756] Initialise system trusted keyrings [ 2.518035] Key type blacklist registered [ 2.519279] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.528491] zbud: loaded [ 2.531038] *** VALIDATE nfs *** [ 2.531880] *** VALIDATE nfs4 *** [ 2.532919] pstore: using deflate compression [ 2.535096] Platform Keyring initialized [ 2.617917] NET: Registered protocol family 38 [ 2.619667] Key type asymmetric registered [ 2.621125] Asymmetric key parser 'x509' registered [ 2.622950] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.625824] io scheduler mq-deadline registered [ 2.627042] io scheduler kyber registered [ 2.628351] io scheduler bfq registered [ 2.629904] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.632337] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.634819] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.636672] ACPI: Power Button [PWRF] [ 2.641598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.648141] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.655798] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.683457] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.711604] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.715034] Non-volatile memory driver v1.3 [ 2.716391] Linux agpgart interface v0.103 [ 2.741143] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.744129] vda: detected capacity change from 0 to 69001216 [ 2.757201] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.760206] vdb: detected capacity change from 0 to 1073741824 [ 2.765415] libphy: Fixed MDIO Bus: probed [ 2.771565] usbcore: registered new interface driver usbserial_generic [ 2.773825] usbserial: USB Serial support registered for generic [ 2.775839] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.778875] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.780598] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.782809] mousedev: PS/2 mouse device common for all mice [ 2.785294] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.788380] rtc_cmos 00:05: RTC can wake from S4 [ 2.790837] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.792040] rtc_cmos 00:05: registered as rtc0 [ 2.796094] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.797454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.798213] intel_pstate: CPU model not supported [ 2.805511] hid: raw HID events driver (C) Jiri Kosina [ 2.807668] usbcore: registered new interface driver usbhid [ 2.809195] usbhid: USB HID core driver [ 2.810338] drop_monitor: Initializing network drop monitor service [ 2.812106] Initializing XFRM netlink socket [ 2.813724] NET: Registered protocol family 10 [ 2.816274] Segment Routing with IPv6 [ 2.817351] NET: Registered protocol family 17 [ 2.819215] mpls_gso: MPLS GSO support [ 2.824217] RAS: Correctable Errors collector initialized. [ 2.825760] AVX version of gcm_enc/dec engaged. [ 2.827218] AES CTR mode by8 optimization enabled [ 2.888128] sched_clock: Marking stable (2888107084, 0)->(3654890381, -766783297) [ 2.891227] registered taskstats version 1 [ 2.892304] Loading compiled-in X.509 certificates [ 2.893770] zswap: loaded using pool lzo/zbud [ 2.913568] Key type big_key registered [ 2.921852] Key type encrypted registered [ 2.923505] ima: No TPM chip found, activating TPM-bypass! [ 2.925807] ima: Allocated hash algorithm: sha1 [ 2.927188] ima: No architecture policies found [ 2.929664] evm: Initialising EVM extended attributes: [ 2.931508] evm: security.selinux [ 2.932376] evm: security.ima [ 2.933243] evm: security.capability [ 2.934161] evm: HMAC attrs: 0x1 [ 2.935964] rtc_cmos 00:05: setting system clock to 2026-03-16 18:01:57 UTC (1773684117) [ 2.940406] debug: unmapping init [mem 0xffffffff8a603000-0xffffffff8a7fffff] [ 2.942827] debug: unmapping init [mem 0xffffffff89382000-0xffffffff89658fff] [ 2.952101] Write protecting the kernel read-only data: 28672k [ 2.954476] debug: unmapping init [mem 0xffffffff87a03000-0xffffffff87bfffff] [ 2.956282] debug: unmapping init [mem 0xffffffff88314000-0xffffffff883fffff] [ 2.983360] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.990701] systemd[1]: Detected virtualization kvm. [ 2.992022] systemd[1]: Detected architecture x86-64. [ 2.993076] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.023319] systemd[1]: No hostname configured. [ 3.024807] systemd[1]: Set hostname to . [ 3.026732] random: systemd: uninitialized urandom read (16 bytes read) [ 3.028660] systemd[1]: Initializing machine ID from random generator. [ 3.146557] random: systemd: uninitialized urandom read (16 bytes read) [ 3.148657] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.152240] random: systemd: uninitialized urandom read (16 bytes read) [ 3.154378] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.157623] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.656816] device-mapper: uevent: version 1.0.3 [ 3.658400] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.264890] virtio_net virtio0 ens2: renamed from eth0 [ 4.291931] scsi host0: ata_piix [ 4.306753] scsi host1: ata_piix [ 4.308169] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.310486] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.032915] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.439132] random: crng init done [ 9.440121] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.713552] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.755967] printk: systemd: 23 output lines suppressed due to ratelimiting [ 10.960685] SELinux: Disabled at runtime. [ 11.011099] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.016797] systemd[1]: Detected virtualization kvm. [ 11.017949] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.414270] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.417494] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.423092] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.427527] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.430716] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.438716] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.443361] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ 11.488188] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.890543] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.175663] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.187370] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.292826] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.325327] EDAC sbridge: Ver: 1.1.2 [ 13.384375] Key type dns_resolver registered [ 13.663666] NFS: Registering the id_resolver key type [ 13.665291] Key type id_resolver registered [ 13.666522] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-client login: [ 37.533120] libcfs: loading out-of-tree module taints kernel. [ 37.563832] Key type ._llcrypt registered [ 37.564796] Key type .llcrypt registered [ 37.828963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 37.835585] alg: No test for adler32 (adler32-zlib) [ 38.837286] Lustre: Lustre: Build Version: 2.17.50_152_gca7da69 [ 39.150890] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 40.767151] Key type lgssc registered [ 41.324037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.013552] Lustre: Mounted lustre-client [ 91.397193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 104.173853] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing check_logdir /tmp/testlogs/ [ 105.682910] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing yml_node [ 107.167941] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 108.011379] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 108.862275] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 109.400304] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 16 14:03:43 EDT 2026 [ 114.622895] LustreError: 7904:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff95c1d8bda000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 114.639672] Lustre: Unmounted lustre-client [ 126.452676] Key type lgssc unregistered [ 126.580845] LNet: 8350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 126.584616] LNetError: 8350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 126.592700] LNet: Removed LNI 192.168.204.42@tcp [ 126.878143] Key type .llcrypt unregistered [ 126.879861] Key type ._llcrypt unregistered [ 135.716760] Key type ._llcrypt registered [ 135.717993] Key type .llcrypt registered [ 135.791163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 135.796792] alg: No test for adler32 (adler32-zlib) [ 136.598849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 136.627312] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 136.636632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 137.944914] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 137.978952] LNet: 9220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 137.982330] LNetError: 9220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 137.999325] LNet: Removed LNI 192.168.204.42@tcp [ 138.226284] Key type .llcrypt unregistered [ 138.227265] Key type ._llcrypt unregistered [ 140.282449] Key type ._llcrypt registered [ 140.283929] Key type .llcrypt registered [ 140.764347] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:04:14 (1773684254) [ 140.877978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 140.884226] alg: No test for adler32 (adler32-zlib) [ 141.690397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 141.720939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 141.729164] LNet: 9637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 141.739220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 141.766950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 141.777639] LNet: 9644:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 144.191417] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:04:18 (1773684258) [ 144.230162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 144.240913] LNet: 10216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.252945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.288401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 146.813538] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:04:20 (1773684260) [ 146.842394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.852151] LNet: 10799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 146.864317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.909355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 149.206929] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:04:23 (1773684263) [ 149.238677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 149.249360] LNet: 11383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 149.260103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 149.295079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 151.753637] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:04:25 (1773684265) [ 151.791720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.808052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.843620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 152.008621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 154.350219] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:04:28 (1773684268) [ 154.394298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 154.403520] LNet: 12559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 154.406347] LNet: 12559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 154.417660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.449519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 154.616695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 154.761400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 157.010265] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:04:31 (1773684271) [ 157.037400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 157.055852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 157.093656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 159.402224] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:04:33 (1773684273) [ 159.427845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.444423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.472898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 159.486623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 159.631321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 159.645880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 159.781499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 159.795941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 159.932344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 159.946731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 160.084206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 160.100952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 160.253607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 160.268135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 160.420512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 160.436386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 160.582314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 160.596897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 160.748417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 160.762433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 160.916448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 160.932720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 161.077267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 161.093957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 161.239907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 161.255712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 161.400897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 161.417878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 163.672794] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:04:37 (1773684277) [ 163.700644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.711144] LNet: 14416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 163.713851] LNet: 14416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 163.726459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.759482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 163.775250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 165.903980] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:04:40 (1773684280) [ 165.937529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.954123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.986324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 165.997899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 168.129200] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:04:42 (1773684282) [ 168.158369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.174463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.208761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 168.224080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 170.464309] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:04:44 (1773684284) [ 170.494361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.511949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.545411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 170.560575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 172.606200] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:04:46 (1773684286) [ 172.633852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.648682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.678566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 172.691102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 174.857323] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:04:48 (1773684288) [ 174.889899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.906480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.939905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 174.956225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 177.195561] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:04:51 (1773684291) [ 177.236361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 177.254964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 177.290600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 177.443935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 179.686941] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:04:53 (1773684293) [ 179.717199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.727429] LNet: 18523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 179.730431] LNet: 18523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 179.739434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.810709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 179.892163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 182.043806] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:04:56 (1773684296) [ 182.074691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 182.091372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 182.141250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 182.214972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.334698] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:04:58 (1773684298) [ 184.362489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.381086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.531240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 184.621216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 186.759545] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:05:00 (1773684300) [ 186.790622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.810747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.839579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 186.855525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 188.920193] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:05:03 (1773684303) [ 188.949917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.964956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.997762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 189.015596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 191.259240] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:05:05 (1773684305) [ 191.295486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.314535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.355517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 193.574358] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:05:07 (1773684307) [ 193.613792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.631823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.666439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 196.029186] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:05:10 (1773684310) [ 196.068993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.085816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.120525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 196.269262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 198.537172] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:05:12 (1773684312) [ 198.571768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.588585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.621184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 198.635992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 200.882779] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:05:14 (1773684314) [ 200.922662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.941679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.972846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 200.986644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 203.028987] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:05:17 (1773684317) [ 203.055144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.071159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.102777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 203.118149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 205.185893] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:05:19 (1773684319) [ 205.214628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.230787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.260339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 205.272648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 207.382662] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:05:21 (1773684321) [ 207.413242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.430177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.459348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 207.483421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 207.490435] LNetError: 25723:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 207.505924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 207.518392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 209.549245] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:05:23 (1773684323) [ 209.575448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.591481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.624881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NgVQYBA2zs [ 209.650774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NgVQYBA2zs [ 209.669889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NgVQYBA2zs [ 211.622381] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:05:25 (1773684325) [ 211.648709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.665832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.691555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 211.704464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 211.722427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 213.616723] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:05:27 (1773684327) [ 213.643568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.652191] LNet: 27473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 213.654489] LNet: 27473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 213.664553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.693248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 214.613224] LNet: Message forwarding will be enabled in 2s [ 217.695113] LNet: Message forwarding enabled [ 217.746429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 217.756062] LNet: Message forwarding will stop in 2s to 6s [ 221.891510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 222.418556] LNet: Message forwarding disabled [ 222.819942] LNet: Message forwarding will be enabled in 2s [ 224.699468] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:05:38 (1773684338) [ 224.728145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.261903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.286649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 227.187881] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:05:41 (1773684341) [ 227.218713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.235796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.262577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 227.275958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 227.290045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 227.301880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 227.313748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 227.325847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 227.335540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 227.347543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 227.363628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 227.376714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 227.389784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 227.403804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 227.417437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 227.431552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 227.446434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 227.460675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 227.477342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 227.492188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 227.504876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 227.512696] LNetError: 28788:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 229.460271] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:05:43 (1773684343) [ 229.489394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.507114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.542477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 231.617662] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:05:45 (1773684345) [ 232.200184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg442-server [ 232.733694] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:05:46 (1773684346) [ 232.757821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.772375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.796571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 232.805058] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 233.314716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[43-47]@tcp [ 233.323570] LNet: 30502:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 235.374867] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:05:49 (1773684349) [ 235.978937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg442-server [ 236.383569] LNetError: 30503:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.46@tcp has gone from up to down [ 236.551709] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:05:50 (1773684350) [ 236.583251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.407197] LNetError: 30290:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.204.43@tcp [ 239.647155] LNetError: 30291:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.47@tcp has gone from up to down [ 239.650458] LNetError: 30291:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 239.653197] LNetError: 31429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 239.666462] LNet: Removed LNI 192.168.204.42@tcp [ 239.674286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.692529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 239.701728] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 239.941374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 239.949281] LNet: 31553:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 239.953148] LNet: 31553:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 239.962227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 242.016502] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:05:56 (1773684356) [ 242.045291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.103906] LNetError: 32138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 243.117558] LNet: Removed LNI 192.168.204.42@tcp [ 243.126546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.160944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 243.183169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.196741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.219539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.239169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.252404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.271213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 243.302388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.317421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.343879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.372111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.386453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.408250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 243.438116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.452950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.479068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.505083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.518311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.538924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 243.568605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.584958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.608248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.636545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.651702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.675258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 243.705403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.720621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.745389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.774463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.789447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.810608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 243.832951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.847229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.873310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.065766] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:06:00 (1773684360) [ 246.706331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg442-server [ 247.273326] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:06:01 (1773684361) [ 247.904550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg442-server [ 248.509499] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:06:02 (1773684362) [ 248.790642] Key type .llcrypt unregistered [ 248.792198] Key type ._llcrypt unregistered [ 250.769445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 250.821403] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 250.823308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 250.825806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 253.013894] Key type ._llcrypt registered [ 253.015727] Key type .llcrypt registered [ 253.086826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 253.092837] alg: No test for adler32 (adler32-zlib) [ 253.897582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.919368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 253.929659] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 254.181020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 254.190138] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 254.897060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 254.905520] LNet: Removed LNI 10.1.2.3@tcp [ 254.948127] LNet: 34264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.951682] LNetError: 34264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 254.968263] LNet: Removed LNI 192.168.204.42@tcp [ 255.204316] Key type .llcrypt unregistered [ 255.205629] Key type ._llcrypt unregistered [ 258.885089] Key type ._llcrypt registered [ 258.886157] Key type .llcrypt registered [ 259.338226] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:06:13 (1773684373) [ 259.496274] Key type .llcrypt unregistered [ 259.497793] Key type ._llcrypt unregistered [ 261.365533] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 261.400777] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 261.405846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 261.408227] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 263.618022] Key type ._llcrypt registered [ 263.619224] Key type .llcrypt registered [ 263.676247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 263.681134] alg: No test for adler32 (adler32-zlib) [ 264.485235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.506510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 264.515536] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 264.740761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 264.748349] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 265.502459] LNet: 35767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 265.504782] LNetError: 35767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 265.507455] LNet: Removed LNI 192.168.204.42@tcp [ 265.513325] LNet: Removed LNI 10.1.2.3@tcp [ 265.723420] Key type .llcrypt unregistered [ 265.724826] Key type ._llcrypt unregistered [ 269.010799] Key type ._llcrypt registered [ 269.012087] Key type .llcrypt registered [ 269.428887] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:06:23 (1773684383) [ 269.589352] Key type .llcrypt unregistered [ 269.590445] Key type ._llcrypt unregistered [ 271.429643] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 271.496623] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 271.498967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 271.501054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 273.696821] Key type ._llcrypt registered [ 273.698015] Key type .llcrypt registered [ 273.763456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 273.770073] alg: No test for adler32 (adler32-zlib) [ 274.575033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.599085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 274.607816] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 275.253131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 275.262431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 275.953643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 275.963704] LNet: Removed LNI 10.1.2.3@tcp [ 275.970906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 275.988321] LNet: Removed LNI 10.1.2.31@tcp [ 275.994872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 276.003319] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 276.249443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 276.256707] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 276.603576] LNet: 37678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.606858] LNetError: 37678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 276.610729] LNet: Removed LNI 10.1.2.3@tcp [ 276.616672] LNet: Removed LNI 10.1.2.31@tcp [ 276.848389] Key type .llcrypt unregistered [ 276.849475] Key type ._llcrypt unregistered [ 280.366752] Key type ._llcrypt registered [ 280.367660] Key type .llcrypt registered [ 280.779753] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:06:34 (1773684394) [ 280.877186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 280.882904] alg: No test for adler32 (adler32-zlib) [ 281.679134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.699125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 281.707122] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 281.957031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.964677] LNet: 38629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 281.966741] LNetError: 38629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 281.980306] LNet: Removed LNI 192.168.204.42@tcp [ 281.987644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.006963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.42@tcp [ 282.016305] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 282.280249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.42@tcp [ 282.293345] LNet: Removed LNI 192.168.204.42@tcp [ 282.524383] Key type .llcrypt unregistered [ 282.525638] Key type ._llcrypt unregistered [ 286.042120] Key type ._llcrypt registered [ 286.043434] Key type .llcrypt registered [ 286.549929] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:06:40 (1773684400) [ 287.219760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg442-server [ 287.839806] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:06:41 (1773684401) [ 288.009439] Key type .llcrypt unregistered [ 288.011155] Key type ._llcrypt unregistered [ 289.985602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 290.031135] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 290.034930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 290.036678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 292.222663] Key type ._llcrypt registered [ 292.223768] Key type .llcrypt registered [ 292.286854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 292.292787] alg: No test for adler32 (adler32-zlib) [ 293.089275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.111454] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 293.347614] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 294.033484] LNet: 40605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.036680] LNetError: 40605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 294.039068] LNet: Removed LNI 192.168.204.42@tcp1 [ 294.044344] LNet: Removed LNI 10.1.2.3@tcp2 [ 294.273467] Key type .llcrypt unregistered [ 294.275050] Key type ._llcrypt unregistered [ 297.627533] Key type ._llcrypt registered [ 297.628777] Key type .llcrypt registered [ 298.044590] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:06:52 (1773684412) [ 298.145748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.150376] alg: No test for adler32 (adler32-zlib) [ 298.943284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.960321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 298.967708] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 299.219305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.225793] LNet: 41571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.227573] LNetError: 41571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.236234] LNet: Removed LNI 192.168.204.42@tcp [ 299.243191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.263611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 299.271996] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 301.188563] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:06:55 (1773684415) [ 301.216053] LNet: 42155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.218555] LNetError: 42155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.224589] LNet: Removed LNI 192.168.204.42@tcp [ 301.433336] Key type .llcrypt unregistered [ 301.435460] Key type ._llcrypt unregistered [ 303.295391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 303.334460] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 303.336390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 303.338340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 305.524831] Key type ._llcrypt registered [ 305.525949] Key type .llcrypt registered [ 305.584418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 305.589795] alg: No test for adler32 (adler32-zlib) [ 306.393462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.413975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 306.422929] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 306.659629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 306.668186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 307.382754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.391616] LNet: 42899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.394192] LNetError: 42899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.396917] LNet: Removed LNI 192.168.204.42@tcp [ 307.402903] LNet: Removed LNI 10.1.2.3@tcp [ 307.410349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.432354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 307.444092] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 307.446778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 307.605683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.614898] LNet: Removed LNI 192.168.204.42@tcp [ 307.625882] LNet: Removed LNI 10.1.2.3@tcp [ 307.633557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.657719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 307.668808] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 307.671292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 307.814382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 307.822059] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 308.161498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.170077] LNet: 43088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.172258] LNet: 43088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 308.175148] LNetError: 43088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.177893] LNetError: 43088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 308.180783] LNet: Removed LNI 192.168.204.42@tcp [ 308.182293] LNet: Removed LNI 10.1.2.3@tcp [ 308.190338] LNet: Removed LNI 192.168.204.42@tcp2 [ 308.198690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.222901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 308.231519] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 308.233328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.236197] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 310.380714] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:07:04 (1773684424) [ 310.406455] LNet: 43676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.409452] LNetError: 43676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.412489] LNetError: 43676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 310.415142] LNet: Removed LNI 192.168.204.42@tcp [ 310.416717] LNet: Removed LNI 10.1.2.3@tcp [ 310.422459] LNet: Removed LNI 192.168.204.42@tcp2 [ 310.636502] Key type .llcrypt unregistered [ 310.637695] Key type ._llcrypt unregistered [ 312.499865] Key type ._llcrypt registered [ 312.501222] Key type .llcrypt registered [ 312.564862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.570322] alg: No test for adler32 (adler32-zlib) [ 313.369102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.388677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 313.396844] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 313.672335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.680123] LNet: 44078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.682868] LNetError: 44078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.690294] LNet: Removed LNI 192.168.204.42@tcp [ 313.697364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.719023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 313.727421] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 313.875041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.890311] LNet: Removed LNI 192.168.204.42@tcp [ 313.898357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.923404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 313.934443] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 314.089105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.106398] LNet: Removed LNI 192.168.204.42@tcp [ 314.115465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.139767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 314.149898] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 314.304204] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 314.332992] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 314.337021] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 314.339647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 316.487840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.495463] LNet: 44216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.497442] LNet: 44216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 316.499762] LNetError: 44216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.501984] LNetError: 44216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 316.512303] LNet: Removed LNI 192.168.204.42@tcp [ 316.520331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.547234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 316.565209] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 318.734991] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:07:12 (1773684432) [ 318.763715] LNet: 44806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.766220] LNetError: 44806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.772490] LNet: Removed LNI 192.168.204.42@tcp [ 318.999967] Key type .llcrypt unregistered [ 319.001594] Key type ._llcrypt unregistered [ 320.861099] Key type ._llcrypt registered [ 320.862678] Key type .llcrypt registered [ 320.923976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.929019] alg: No test for adler32 (adler32-zlib) [ 321.735714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.756097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 321.765244] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 322.055504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.063694] LNet: 45210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.066600] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.077377] LNet: Removed LNI 192.168.204.42@tcp [ 322.085477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.109954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 322.119052] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 324.156916] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:07:18 (1773684438) [ 324.183534] LNet: 45794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.185560] LNetError: 45794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.192393] LNet: Removed LNI 192.168.204.42@tcp [ 324.402668] Key type .llcrypt unregistered [ 324.403775] Key type ._llcrypt unregistered [ 326.135996] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 326.203981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 326.207929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 326.209836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 328.370715] Key type ._llcrypt registered [ 328.371717] Key type .llcrypt registered [ 328.431478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 328.436619] alg: No test for adler32 (adler32-zlib) [ 329.238351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.260574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 329.273690] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 329.556185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.562435] LNet: 46250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.564712] LNetError: 46250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.571257] LNet: Removed LNI 192.168.204.42@tcp [ 329.578595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.600396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 329.610330] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 331.651902] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:07:25 (1773684445) [ 331.681350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.692260] LNet: 46842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.695176] LNetError: 46842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.706352] LNet: Removed LNI 192.168.204.42@tcp [ 331.716528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.751865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 333.633682] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:07:27 (1773684447) [ 333.662698] LNet: 47421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.869560] Key type .llcrypt unregistered [ 333.871250] Key type ._llcrypt unregistered [ 335.680446] Key type ._llcrypt registered [ 335.681730] Key type .llcrypt registered [ 335.747208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.752164] alg: No test for adler32 (adler32-zlib) [ 336.553317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.572546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 336.581876] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 336.855533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.862358] LNet: 47826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.864359] LNetError: 47826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.877285] LNet: Removed LNI 192.168.204.42@tcp [ 336.884613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.903891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 336.913522] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 338.871716] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:07:33 (1773684453) [ 338.896553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.904989] LNet: 48414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.907436] LNetError: 48414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.915588] LNet: Removed LNI 192.168.204.42@tcp [ 338.925278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.950345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 340.857119] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:07:34 (1773684454) [ 341.341918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 341.840541] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:07:35 (1773684455) [ 342.326772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 342.874310] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:07:36 (1773684456) [ 343.402143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 343.960552] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:07:38 (1773684458) [ 344.480237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 345.007688] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:07:39 (1773684459) [ 345.509996] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 346.073396] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:07:40 (1773684460) [ 346.110115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.119390] LNet: 50466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.129769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.153855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 346.162878] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 346.435126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.42 [ 346.444038] LNet: 50603:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 346.448268] LNet: Removed LNI 192.168.204.42@tcp [ 348.574155] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:07:42 (1773684462) [ 348.603062] LNet: 51167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.606067] LNetError: 51167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.615699] LNet: Removed LNI 192.168.204.42@tcp [ 348.857752] Key type .llcrypt unregistered [ 348.859207] Key type ._llcrypt unregistered [ 350.808145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 350.914222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 350.915933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 350.917991] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 353.193672] Key type ._llcrypt registered [ 353.194833] Key type .llcrypt registered [ 353.253399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 353.259452] alg: No test for adler32 (adler32-zlib) [ 354.206578] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 356.374781] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:07:50 (1773684470) [ 356.404112] LNet: 52070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.406847] LNetError: 52070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 356.415693] LNet: Removed LNI 10.1.2.3@tcp [ 356.638459] Key type .llcrypt unregistered [ 356.639738] Key type ._llcrypt unregistered [ 358.426233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 358.523487] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 358.525130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 358.527285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 360.789540] Key type ._llcrypt registered [ 360.790750] Key type .llcrypt registered [ 360.845575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 360.851154] alg: No test for adler32 (adler32-zlib) [ 361.670488] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 363.772819] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:07:57 (1773684477) [ 363.801300] LNet: 52960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.804730] LNetError: 52960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 363.816449] LNet: Removed LNI 10.1.2.3@tcp [ 364.056388] Key type .llcrypt unregistered [ 364.057306] Key type ._llcrypt unregistered [ 366.005271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 366.099403] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 366.101217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 366.103410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 368.382786] Key type ._llcrypt registered [ 368.384083] Key type .llcrypt registered [ 368.453695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 368.460107] alg: No test for adler32 (adler32-zlib) [ 369.279380] LNet: 53267:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 369.283844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 371.450934] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:08:05 (1773684485) [ 371.484189] LNet: 53852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.487237] LNetError: 53852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 371.496718] LNet: Removed LNI 10.1.2.3@tcp [ 371.730464] Key type .llcrypt unregistered [ 371.731536] Key type ._llcrypt unregistered [ 373.601479] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 373.694940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 373.697205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 373.699977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 375.956526] Key type ._llcrypt registered [ 375.957525] Key type .llcrypt registered [ 376.016662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 376.021796] alg: No test for adler32 (adler32-zlib) [ 376.823134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.853747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 379.462384] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:08:13 (1773684493) [ 379.587408] LNet: 55086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.589649] LNetError: 55086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 379.603598] LNet: Removed LNI 10.1.2.3@tcp [ 379.809508] Key type .llcrypt unregistered [ 379.810520] Key type ._llcrypt unregistered [ 381.623499] Key type ._llcrypt registered [ 381.625133] Key type .llcrypt registered [ 381.691722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 381.698099] alg: No test for adler32 (adler32-zlib) [ 382.500267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 382.513752] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 385.317112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 385.539976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 385.554990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 385.569732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 385.577732] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 385.651708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 385.662180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 385.673201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 385.685522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 385.697815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 385.710306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 385.720675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 385.732474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 385.740581] LNet: 55522:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 385.747841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 385.759376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 385.771983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 385.784071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 385.797472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 385.811082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 385.823991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 385.985728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 385.999991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 386.012785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 386.088548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 386.100494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 386.111975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 386.123308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 386.136433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 386.148595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 386.160079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 386.173213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 386.185697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 386.198914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 386.211355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 386.224452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 386.236471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 386.249708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 386.263636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 386.432732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 386.445744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 386.458083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 386.465674] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 386.467978] LNet: Skipped 1 previous similar message [ 386.536671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 386.548173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 386.559553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 386.569254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 386.581559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 386.591208] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 386.594970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 386.607564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 386.620805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 386.628452] LNet: 55740:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 386.631022] LNet: 55740:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 386.637847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 386.649952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 386.661404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 386.671458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 386.683739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 386.697427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 386.710291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 386.878481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 386.891840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 386.906410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 386.985564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 386.997530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 387.008888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 387.019926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 387.030340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 387.041061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 387.054047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 387.064206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 387.076386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 387.088812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 387.098974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 387.107957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 387.118810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 387.129596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 387.139969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 387.289330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 387.300369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 387.311686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 387.387381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 387.396901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 387.406635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 387.416380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 387.425940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 387.435531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 387.447831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 387.459790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 387.471200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 387.482440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 387.496108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 387.508109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 387.519685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 387.531598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 387.543373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 387.697388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 387.708929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 387.720378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 387.729405] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 387.731417] LNet: Skipped 3 previous similar messages [ 387.799428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 387.808667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 387.819325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 387.829816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 387.841406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 387.854411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 387.865634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 387.877351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 387.883671] LNet: 56067:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 387.886479] LNet: 56067:0:(api-ni.c:299:recovery_interval_set()) Skipped 2 previous similar messages [ 387.894208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 387.904516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 387.914866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 387.927520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 387.941090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 387.952339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 387.966545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 388.046524] LNet: 56108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.048638] LNetError: 56108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.056287] LNet: Removed LNI 192.168.204.42@tcp [ 388.195351] Key type .llcrypt unregistered [ 388.196633] Key type ._llcrypt unregistered [ 393.036028] Key type ._llcrypt registered [ 393.037402] Key type .llcrypt registered [ 393.471114] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:08:27 (1773684507) [ 393.725266] Key type .llcrypt unregistered [ 393.726451] Key type ._llcrypt unregistered [ 395.490278] Key type ._llcrypt registered [ 395.492189] Key type .llcrypt registered [ 395.553550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 395.558176] alg: No test for adler32 (adler32-zlib) [ 396.357354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 396.368211] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 399.144989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 402.260462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.273628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 402.282191] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 402.734800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 402.749731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 402.764892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 402.778632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 402.792093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 402.804103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 402.817118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 402.832316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 402.846906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 402.857459] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 402.859677] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 402.927683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 402.941324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 402.954722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 402.967402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 402.978851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 402.988893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 402.999692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 403.012278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 403.020254] LNet: 57569:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 403.029925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 403.042405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 403.053613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 403.066360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 403.077458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 403.088603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 403.099255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 403.273887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 403.288820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 403.303931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 403.320299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 403.336274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 403.351824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 403.368340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 403.385134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 403.400363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 403.412409] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 403.415544] LNet: Skipped 2 previous similar messages [ 403.485836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 403.496721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 403.506566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 403.518100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 403.529905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 403.541176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 403.552643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 403.562720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 403.568997] LNet: 57690:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 403.575480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 403.586451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 403.596548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 403.607169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 403.619648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 403.631730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 403.642748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 403.811332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 403.823177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 403.835965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 403.848192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 403.862994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 403.878565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 403.893663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 403.908371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 403.923290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 403.996533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 404.007131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 404.019645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 404.032241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 404.045296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 404.057665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 404.072453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 404.084220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 404.096828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 404.109957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 404.121316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 404.133087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 404.147256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 404.161269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 404.176600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 404.347956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 404.362975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 404.378308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 404.392511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 404.407684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 404.423736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 404.440243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 404.455247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 404.470433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 404.479438] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 404.482163] LNet: Skipped 6 previous similar messages [ 404.551689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 404.561910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 404.572136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 404.582164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 404.592843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 404.604571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 404.618294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 404.629497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 404.636628] LNet: 57932:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 404.639576] LNet: 57932:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 404.646780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 404.657852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 404.668522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 404.679624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 404.688970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 404.699715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 404.710807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 404.887410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 404.901641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 404.914607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 404.929058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 404.944467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 404.959209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 404.973885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 404.989254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 405.003555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 405.076700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 405.087859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 405.099603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 405.111911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 405.123676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 405.135868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 405.149767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 405.161687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 405.173100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 405.184232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 405.195923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 405.206316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 405.215685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 405.227236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 405.239198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 405.401801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 405.414957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 405.430941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 405.447653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 405.463572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 405.479675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 405.496027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 405.511079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 405.523495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 405.604316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 405.615482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 405.626874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 405.638868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 405.650118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 405.662504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 405.677059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 405.690378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 405.704071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 405.716696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 405.730546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 405.742848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 405.756512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 405.772332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 405.786978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 407.914317] LNet: 58250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.917197] LNetError: 58250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 407.920918] LNet: Removed LNI 192.168.204.42@tcp [ 408.933383] LNet: Removed LNI 192.168.204.42@tcp1 [ 409.089579] Key type .llcrypt unregistered [ 409.090446] Key type ._llcrypt unregistered [ 414.265373] Key type ._llcrypt registered [ 414.266872] Key type .llcrypt registered [ 414.720558] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:08:48 (1773684528) [ 414.951346] Key type .llcrypt unregistered [ 414.952375] Key type ._llcrypt unregistered [ 416.860440] Key type ._llcrypt registered [ 416.861720] Key type .llcrypt registered [ 416.934666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 416.940611] alg: No test for adler32 (adler32-zlib) [ 417.740976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 417.752266] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 420.590554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 420.790790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 420.803709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 420.819019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 420.827473] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 420.896439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 420.907337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 420.918360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 420.930484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 420.942991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 420.957311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 420.970318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 420.982559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 420.990298] LNet: 59495:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 420.997734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 421.009720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 421.022105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 421.032623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 421.043574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.056492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.069708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 421.238117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 421.253636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 421.267315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 421.347611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 421.359324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 421.370808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 421.385793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 421.397265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 421.406923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 421.418558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 421.430600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 421.442123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 421.453925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 421.465605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 421.478789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 421.492224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.506801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.517778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 421.689046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 421.705858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 421.719973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 421.727573] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 421.729438] LNet: Skipped 1 previous similar message [ 421.800462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 421.812497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 421.827968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 421.843183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 421.859380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 421.877282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 421.892069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 421.907685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 421.916057] LNet: 59713:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 421.919177] LNet: 59713:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 421.927022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 421.940744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 421.955435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 421.966856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 421.980373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.995149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 422.009987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 422.104310] LNet: 59754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.107068] LNetError: 59754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.115528] LNet: Removed LNI 192.168.204.42@tcp [ 422.255381] Key type .llcrypt unregistered [ 422.256580] Key type ._llcrypt unregistered [ 427.100256] Key type ._llcrypt registered [ 427.101364] Key type .llcrypt registered [ 427.517040] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:09:01 (1773684541) [ 427.761323] Key type .llcrypt unregistered [ 427.763147] Key type ._llcrypt unregistered [ 429.484385] Key type ._llcrypt registered [ 429.485285] Key type .llcrypt registered [ 429.546879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 429.552685] alg: No test for adler32 (adler32-zlib) [ 430.348536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 430.357166] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 432.972706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 435.360545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.373875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 435.383562] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 435.827788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 435.840599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 435.855765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 435.868565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 435.880962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 435.897607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 435.913824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 435.926084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 435.938464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 435.944229] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 435.946762] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 436.011867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 436.024418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 436.036175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 436.045216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 436.058246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 436.070888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 436.084441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 436.096651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 436.103716] LNet: 61216:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 436.110482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 436.121250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.132119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 436.141800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 436.152074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.163834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.175614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 436.191305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.350733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 436.362927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 436.374930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 436.386492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 436.398533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 436.411309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 436.425549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 436.440077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 436.455291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 436.463181] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 436.465169] LNet: Skipped 2 previous similar messages [ 436.527779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 436.537099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 436.550471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 436.561503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 436.572622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 436.582914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 436.595160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 436.608865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 436.617358] LNet: 61339:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 436.626751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 436.638554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.648820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 436.660662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 436.671496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.683075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.695140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 436.707764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.865118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 436.878702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 436.889974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 436.900851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 436.910889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 436.921103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 436.934057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 436.945676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 436.959395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 437.031500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 437.042030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 437.052708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 437.063431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 437.074052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 437.085582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 437.098401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 437.111691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 437.124521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 437.135887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 437.147463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 437.158873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 437.169352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 437.180465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 437.189647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 437.199720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 439.191115] LNet: 61539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.193241] LNetError: 61539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.195431] LNet: Removed LNI 192.168.204.42@tcp [ 439.210279] LNet: Removed LNI 192.168.204.42@tcp1 [ 439.336611] Key type .llcrypt unregistered [ 439.337782] Key type ._llcrypt unregistered [ 443.995684] Key type ._llcrypt registered [ 443.996657] Key type .llcrypt registered [ 444.410295] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:09:18 (1773684558) [ 444.551439] Key type .llcrypt unregistered [ 444.552459] Key type ._llcrypt unregistered [ 446.272265] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 446.324879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 446.328133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 446.329843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 448.515673] Key type ._llcrypt registered [ 448.516670] Key type .llcrypt registered [ 448.573085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 448.578899] alg: No test for adler32 (adler32-zlib) [ 449.381493] LNet: 62614:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 449.385880] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 449.397836] LNet: 62628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.400214] LNetError: 62628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.410300] LNet: Removed LNI 192.168.204.42@tcp [ 449.621392] Key type .llcrypt unregistered [ 449.622369] Key type ._llcrypt unregistered [ 451.421868] Key type ._llcrypt registered [ 451.423015] Key type .llcrypt registered [ 451.478236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 451.483241] alg: No test for adler32 (adler32-zlib) [ 452.280689] LNet: 62887:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 452.285564] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 452.287112] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 452.298773] LNet: 62901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.301203] LNetError: 62901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.303094] LNet: Removed LNI 192.168.204.42@tcp [ 452.311283] LNet: Removed LNI 10.1.2.3@tcp1 [ 452.529592] Key type .llcrypt unregistered [ 452.531270] Key type ._llcrypt unregistered [ 454.298663] Key type ._llcrypt registered [ 454.299478] Key type .llcrypt registered [ 454.348151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 454.352969] alg: No test for adler32 (adler32-zlib) [ 455.152728] LNet: 63159:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 455.156522] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 455.157714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 455.170777] LNet: 63173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.173293] LNetError: 63173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.176575] LNet: Removed LNI 192.168.204.42@tcp [ 455.183321] LNet: Removed LNI 10.1.2.3@tcp [ 455.395424] Key type .llcrypt unregistered [ 455.396598] Key type ._llcrypt unregistered [ 457.258423] Key type ._llcrypt registered [ 457.259420] Key type .llcrypt registered [ 457.319319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 457.325560] alg: No test for adler32 (adler32-zlib) [ 458.136397] LNet: 63432:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 458.141996] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 458.143960] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 458.159241] LNet: 63446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.161621] LNetError: 63446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.164856] LNet: Removed LNI 192.168.204.42@tcp [ 458.179297] LNet: Removed LNI 10.1.2.3@tcp [ 458.402461] Key type .llcrypt unregistered [ 458.404074] Key type ._llcrypt unregistered [ 460.314355] Key type ._llcrypt registered [ 460.315438] Key type .llcrypt registered [ 460.374780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 460.381385] alg: No test for adler32 (adler32-zlib) [ 461.181150] LNet: 63710:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 461.185391] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 461.187322] LNetError: 63710:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 461.189383] LNetError: Error -100 starting up LNI tcp [ 461.196281] LNet: Removed LNI 192.168.204.42@tcp [ 461.426343] Key type .llcrypt unregistered [ 461.427426] Key type ._llcrypt unregistered [ 464.947783] Key type ._llcrypt registered [ 464.949255] Key type .llcrypt registered [ 465.383352] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:09:39 (1773684579) [ 465.632377] Key type .llcrypt unregistered [ 465.633282] Key type ._llcrypt unregistered [ 467.411058] Key type ._llcrypt registered [ 467.412226] Key type .llcrypt registered [ 467.472266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 467.476384] alg: No test for adler32 (adler32-zlib) [ 468.276679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 468.286483] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 470.987440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 471.203295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 471.218745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 471.232054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 471.240033] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 471.313039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 471.324077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 471.336412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 471.348772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 471.361300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 471.371990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 471.381736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 471.391100] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 471.392760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 471.401566] LNet: 64962:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 471.409429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 471.422286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 471.434164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 471.445930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 471.459514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 471.473845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 471.564849] LNet: 65001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.567873] LNetError: 65001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.576412] LNet: Removed LNI 192.168.204.42@tcp [ 471.727406] Key type .llcrypt unregistered [ 471.728490] Key type ._llcrypt unregistered [ 477.070905] Key type ._llcrypt registered [ 477.072125] Key type .llcrypt registered [ 477.138565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 477.144298] alg: No test for adler32 (adler32-zlib) [ 477.953734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 477.964314] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 480.804129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 483.415036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.432480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 483.442186] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 483.895599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 483.909150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 483.919746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 483.930815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 483.945403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 483.960956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 483.979973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 483.997252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 484.014093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 484.023937] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 484.097828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 484.108949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 484.121696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 484.127097] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 484.128982] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 484.136599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 484.150200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 484.162359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 484.176939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 484.189684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 484.198327] LNet: 65876:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 484.206350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 484.218258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 484.229170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 484.243657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 484.256426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 484.269124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 486.215198] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 486.312293] LNet: 65950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.315794] LNetError: 65950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 486.318760] LNet: Removed LNI 192.168.204.42@tcp [ 486.326323] LNet: Removed LNI 192.168.204.42@tcp1 [ 486.461564] Key type .llcrypt unregistered [ 486.463096] Key type ._llcrypt unregistered [ 491.309310] Key type ._llcrypt registered [ 491.310157] Key type .llcrypt registered [ 491.725505] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:10:05 (1773684605) [ 491.835808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 491.840495] alg: No test for adler32 (adler32-zlib) [ 492.639265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.659066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 492.667228] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 492.898879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 492.905582] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 493.185648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 493.210826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 493.226391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 493.241431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 493.254625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 493.267966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 493.282455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 493.289804] LNetError: 67047:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 493.663241] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 493.663266] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 495.711371] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 496.426478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 499.807293] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 500.606483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 500.618387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 500.638284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.644762] LNet: 67164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.647604] LNetError: 67164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.649499] LNet: Removed LNI 192.168.204.42@tcp [ 500.656256] LNet: Removed LNI 192.168.204.42@tcp1 [ 500.662593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.680962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 500.688939] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 500.902918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 500.910703] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 501.178625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 501.191206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 501.213875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 501.229645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 501.243689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 501.257597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 501.273651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 501.286417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 501.294678] LNetError: 67436:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 503.775325] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 503.777498] LNet: Skipped 1 previous similar message [ 504.434676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 507.871323] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 512.719000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 512.732283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 512.746990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 514.804993] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:10:28 (1773684628) [ 514.830980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.839704] LNet: 68157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.842135] LNetError: 68157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.844086] LNetError: 68157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 514.846352] LNet: Removed LNI 192.168.204.42@tcp [ 514.852098] LNet: Removed LNI 192.168.204.42@tcp1 [ 514.860152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.878914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 514.886099] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 515.102753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 515.110644] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 515.370400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 515.394556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 515.405699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 515.416856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 515.427583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 515.438834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 515.450666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 515.483695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 515.871101] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.42@tcp [ 516.895260] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 516.897804] LNet: Skipped 1 previous similar message [ 530.863375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 530.909501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 530.920939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 530.932155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 500 [ 530.960813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 537.129105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 537.150679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.157526] LNet: 68661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.159723] LNetError: 68661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.161740] LNetError: 68661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 537.163990] LNet: Removed LNI 192.168.204.42@tcp [ 537.170261] LNet: Removed LNI 192.168.204.42@tcp1 [ 537.177544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.196292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 537.204633] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 537.436125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 537.446273] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 537.716822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 537.730443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 537.752107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 537.765217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 537.777457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 537.789898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 537.801369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 537.812658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 538.207288] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 538.209107] LNet: Skipped 3 previous similar messages [ 538.904079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 549.230981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 549.244601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 549.258871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 551.234779] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:11:05 (1773684665) [ 551.362907] LNet: 69716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.364915] LNetError: 69716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.366546] LNetError: 69716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 551.369623] LNet: Removed LNI 192.168.204.42@tcp [ 551.374514] LNet: Removed LNI 192.168.204.42@tcp1 [ 551.588678] Key type .llcrypt unregistered [ 551.590589] Key type ._llcrypt unregistered [ 553.330651] Key type ._llcrypt registered [ 553.332150] Key type .llcrypt registered [ 553.390947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 553.396172] alg: No test for adler32 (adler32-zlib) [ 554.190548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 554.199229] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 557.242181] Lustre: DEBUG MARKER: Initial discovery [ 557.259130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 557.998821] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 558.019156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 558.035405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 558.043613] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 558.054729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 558.066728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -l 3 [ 558.078405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -e local_error [ 558.091234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m PUT -l 6 [ 558.303189] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 558.535151] Lustre: DEBUG MARKER: Force oleg442-client.virtnet to discover 192.168.204.142@tcp (in background) [ 558.557284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 558.992664] Lustre: DEBUG MARKER: Force oleg442-server to discover 192.168.204.42@tcp [ 559.702586] Lustre: DEBUG MARKER: Wait for 70468 [ 561.631209] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 562.080659] Lustre: DEBUG MARKER: Finished wait on 70468 [ 562.099868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 562.114085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 562.198051] LNet: 70925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.199805] LNetError: 70925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.207250] LNet: Removed LNI 192.168.204.42@tcp [ 562.342504] Key type .llcrypt unregistered [ 562.344261] Key type ._llcrypt unregistered [ 566.933695] Key type ._llcrypt registered [ 566.935735] Key type .llcrypt registered [ 567.356503] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:11:21 (1773684681) [ 567.512346] Key type .llcrypt unregistered [ 567.513270] Key type ._llcrypt unregistered [ 569.217669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 569.261346] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 569.263093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 569.265442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 571.454516] Key type ._llcrypt registered [ 571.455844] Key type .llcrypt registered [ 571.510457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 571.515571] alg: No test for adler32 (adler32-zlib) [ 572.312946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.334436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 572.344756] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 572.585082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 572.592289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 575.166622] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:11:29 (1773684689) [ 575.202339] LNet: 72994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.204668] LNetError: 72994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.208576] LNet: Removed LNI 192.168.204.42@tcp [ 575.214681] LNet: Removed LNI 10.1.2.3@tcp [ 575.435643] Key type .llcrypt unregistered [ 575.437875] Key type ._llcrypt unregistered [ 577.341410] Key type ._llcrypt registered [ 577.342732] Key type .llcrypt registered [ 577.401868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.407058] alg: No test for adler32 (adler32-zlib) [ 578.206430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.227163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.235754] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 578.465566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 578.472792] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 578.785919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.42@tcp [ 579.807147] LNetError: Unexpected error -22 connecting to 192.168.204.42@tcp at host 192.168.204.42:988 [ 580.745212] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:11:34 (1773684694) [ 580.774282] LNet: 74122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.776722] LNetError: 74122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.779860] LNet: Removed LNI 192.168.204.42@tcp [ 580.787553] LNet: Removed LNI 10.1.2.3@tcp [ 581.011365] Key type .llcrypt unregistered [ 581.012370] Key type ._llcrypt unregistered [ 582.840894] Key type ._llcrypt registered [ 582.842017] Key type .llcrypt registered [ 582.917649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.924758] alg: No test for adler32 (adler32-zlib) [ 583.724961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.748927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 583.757544] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 583.998522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 584.005026] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 584.293840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.42@tcp1 --nid 192.168.204.42@tcp2 [ 585.551781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp2 [ 585.559668] LNetError: 74978:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.42@tcp1 to 192.168.204.42@tcp2 - Route table may be misconfigured [ 585.566721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp2 192.168.204.42@tcp1 [ 587.461834] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:11:41 (1773684701) [ 587.490055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.499787] LNet: 75545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.502732] LNetError: 75545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.505580] LNet: Removed LNI 192.168.204.42@tcp1 [ 588.518215] LNet: Removed LNI 192.168.204.42@tcp2 [ 588.529934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.555526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 588.565981] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 588.811570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 588.822337] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 589.110542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 589.123944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.42@tcp [ 589.135588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp1 [ 589.150623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp -e network_timeout [ 589.166682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -e network_timeout [ 589.183594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -e network_timeout [ 589.202654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -e network_timeout [ 589.216403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 589.223222] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 589.229991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 589.262670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 589.294565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 589.599139] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 591.264381] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:11:45 (1773684705) [ 591.289469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.299593] LNet: 76405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.302800] LNetError: 76405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.305396] LNetError: 76405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 591.307495] LNet: Removed LNI 192.168.204.42@tcp [ 592.357525] LNet: Removed LNI 192.168.204.42@tcp1 [ 592.367871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.398067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 592.480390] LNet: 76448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.610165] Key type .llcrypt unregistered [ 592.611228] Key type ._llcrypt unregistered [ 595.913385] Key type ._llcrypt registered [ 595.914659] Key type .llcrypt registered [ 596.330596] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:11:50 (1773684710) [ 596.473525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.480091] alg: No test for adler32 (adler32-zlib) [ 597.272971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.707874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 598.240970] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:11:52 (1773684712) [ 598.266590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.276888] LNet: 77539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.287055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.310945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 598.319858] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 598.532607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 598.539623] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 598.804899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 598.816185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp1 [ 598.828111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp1 [ 600.758123] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:11:54 (1773684714) [ 601.306783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg442-server [ 601.824930] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:11:55 (1773684715) [ 602.375353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg442-server [ 602.871565] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:11:57 (1773684717) [ 603.422199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg442-server [ 603.920740] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:11:58 (1773684718) [ 604.458236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg442-server [ 604.971858] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:11:59 (1773684719) [ 605.543457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg442-server [ 606.072101] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:12:00 (1773684720) [ 606.636496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg442-server [ 607.150663] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:12:01 (1773684721) [ 607.729178] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg442-server [ 608.253743] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:12:02 (1773684722) [ 608.825667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg442-server [ 609.341262] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:12:03 (1773684723) [ 609.912146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg442-server [ 610.424346] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:12:04 (1773684724) [ 610.459148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.467357] LNet: 81412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.469765] LNetError: 81412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.471897] LNet: Removed LNI 192.168.204.42@tcp [ 611.493434] LNet: Removed LNI 192.168.204.42@tcp1 [ 611.502133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.524672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.534773] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 611.785488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 611.814234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 611.877888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.885852] LNet: 81565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.888679] LNetError: 81565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.891702] LNetError: 81565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 611.904324] LNet: Removed LNI 192.168.204.42@tcp [ 611.910868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.929959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 611.938714] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 612.188161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 612.244772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.285346] LNet: Removed LNI 192.168.204.42@tcp [ 613.293561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.314089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.322869] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 613.569538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 613.601448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 613.674158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.682629] LNetError: 81857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.685298] LNetError: 81857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 613.692438] LNet: Removed LNI 192.168.204.42@tcp [ 613.700878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.725123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 613.736151] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 614.010860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 614.069258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.075539] LNet: 82002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.078783] LNet: 82002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 615.141279] LNet: Removed LNI 192.168.204.42@tcp [ 615.148676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.169117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.178616] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 615.412788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 615.439223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 615.498603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.549327] LNet: Removed LNI 192.168.204.42@tcp [ 616.556300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.576146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 616.584659] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 616.832899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 616.896190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.902581] LNetError: 82294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.904499] LNetError: 82294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 616.911479] LNet: Removed LNI 192.168.204.42@tcp [ 616.918953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.941383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 616.948564] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 617.168360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 617.192427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 617.253683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.281280] LNet: Removed LNI 192.168.204.42@tcp [ 618.288795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.305641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 618.313968] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 618.554389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 618.621583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.629128] LNet: 82587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.631307] LNet: 82587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 619.685280] LNet: Removed LNI 192.168.204.42@tcp [ 619.692679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.711476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.720852] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 619.945188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 619.971379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 621.093564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.102915] LNetError: 82743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.105707] LNetError: 82743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 621.115311] LNet: Removed LNI 192.168.204.42@tcp [ 621.123573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.144788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 621.152995] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 621.412988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 622.519487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.589323] LNet: Removed LNI 192.168.204.42@tcp [ 623.597906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.617647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.627629] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 623.880789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 623.912690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 625.032990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.085271] LNet: Removed LNI 192.168.204.42@tcp [ 626.092350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.111526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 626.121573] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 626.379724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 626.463116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.493287] LNet: Removed LNI 192.168.204.42@tcp [ 627.501074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.524474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.533804] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 627.776675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 627.798644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.805530] LNet: 83331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.807265] LNet: 83331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 627.815339] LNet: Removed LNI 192.168.204.42@tcp [ 627.823745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.844727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.854254] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 628.114692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 630.292468] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:12:24 (1773684744) [ 630.320951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.329033] LNetError: 84028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.330860] LNetError: 84028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 630.338397] LNet: Removed LNI 192.168.204.42@tcp [ 630.346487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.368651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 630.378881] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 630.765815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.779417] LNet: Removed LNI 192.168.204.42@tcp231 [ 630.786976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.809780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 630.819584] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 630.822179] LNet: 84180:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 630.942251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 630.960413] LNet: Removed LNI 192.168.204.42@tcp231 [ 630.967419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 630.975915] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 631.354151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.371468] LNet: Removed LNI 192.168.204.42@tcp231 [ 631.378315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.403341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 631.414574] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 631.417380] LNet: 84333:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 631.538979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 631.554479] LNet: Removed LNI 192.168.204.42@tcp231 [ 631.561630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 631.572292] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 631.918807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.932251] LNet: Removed LNI 192.168.204.42@tcp231 [ 631.939121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.964271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 631.974446] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 632.110459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 632.126610] LNet: Removed LNI 192.168.204.42@tcp231 [ 632.132369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 632.141257] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 632.533893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.546425] LNet: Removed LNI 192.168.204.42@tcp231 [ 632.553561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.577988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 632.587319] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 632.589477] LNet: 84635:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.591926] LNet: 84635:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 632.715898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 632.734368] LNet: Removed LNI 192.168.204.42@tcp231 [ 632.741138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 632.751612] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 635.080118] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:12:29 (1773684749) [ 635.106956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.122431] LNet: Removed LNI 192.168.204.42@tcp231 [ 635.130398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.155138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 635.161936] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 635.404198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 635.420754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 635.442308] LNet: Removed LNI 192.168.204.42@tcp232 [ 635.449914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 635.461490] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 635.709186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 635.725459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 635.741411] LNet: Removed LNI 192.168.204.42@tcp232 [ 635.748025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 635.756389] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 635.990105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 636.002788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 636.017451] LNet: Removed LNI 192.168.204.42@tcp232 [ 636.022783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 636.031308] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 636.267088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 636.282050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 636.300352] LNet: Removed LNI 192.168.204.42@tcp232 [ 636.306557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 638.256133] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:12:32 (1773684752) [ 638.580023] Key type .llcrypt unregistered [ 638.580890] Key type ._llcrypt unregistered [ 640.458332] Key type ._llcrypt registered [ 640.459500] Key type .llcrypt registered [ 640.526168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.531576] alg: No test for adler32 (adler32-zlib) [ 641.340752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 641.352808] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 644.168985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 646.630774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.644820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 646.655435] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 647.109643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 647.125397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 647.138854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 647.154160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 647.166826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 647.184688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 647.202137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 647.216242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 647.228849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 647.235937] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 647.238133] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 647.316135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 647.328112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 647.338183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 647.347745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 647.356493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 647.365932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 647.377077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 647.389433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 647.396068] LNet: 87130:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 647.403629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 647.414467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.427036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 647.440447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 647.453589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.467768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 647.633196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 647.645896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 647.658814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 647.673329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 647.685942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 647.697483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 647.708632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 647.721952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 647.737185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 647.748034] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 647.824636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 647.835614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 647.847673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 647.858802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 647.872317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 647.887464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 647.901916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 647.917697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 647.926366] LNet: 87250:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 647.934357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 647.948931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.961727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 647.973941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 647.986446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.998753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 648.168602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 648.181402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 648.195290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 648.207984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 648.220929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 648.235057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 648.249134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 648.262634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 648.274850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 648.360089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 648.371826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 648.384173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 648.397917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 648.413593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 648.425890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 648.440090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 648.452190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 648.462996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 648.476754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 648.488819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 648.502575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 648.514445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 648.525818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 648.693396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 648.708882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 648.722726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 648.737709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 648.755434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 648.766951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 648.778891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 648.790247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 648.800866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 648.807764] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 648.809932] LNet: Skipped 2 previous similar messages [ 648.887916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 648.897289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 648.908069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 648.918482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 648.929166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 648.939112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 648.950163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 648.959550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 648.965537] LNet: 87490:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 648.967753] LNet: 87490:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 648.974052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 648.984510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 648.997704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 649.012323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 649.022252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 649.033805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.200832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 649.216065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 649.228910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 649.243298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 649.255203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 649.269182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 649.287603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 649.301137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 649.316658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 649.398655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 649.408828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 649.419260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 649.432316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 649.443705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 649.456907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 649.468141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 649.480303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 649.491275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 649.501887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 649.510832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 649.520481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 649.533097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 649.547111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.717474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 649.730878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 649.744623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 649.758701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 649.773743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 649.791031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 649.804355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 649.821095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 649.835809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 649.844211] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 649.918257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 649.931770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 649.943895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 649.955959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 649.967402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 649.980240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 649.994702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 650.008853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 650.023807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 650.036599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 650.049185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 650.059881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 650.071403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 650.084884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 652.200071] LNet: 87804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.204135] LNetError: 87804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.207957] LNet: Removed LNI 192.168.204.42@tcp [ 652.217478] LNet: Removed LNI 192.168.204.42@tcp1 [ 652.350696] Key type .llcrypt unregistered [ 652.351807] Key type ._llcrypt unregistered [ 657.113876] Key type ._llcrypt registered [ 657.115164] Key type .llcrypt registered [ 657.530987] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:12:51 (1773684771) [ 657.640487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.645183] alg: No test for adler32 (adler32-zlib) [ 658.444770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.461301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.470065] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 658.734646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 658.743116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 658.749592] LNetError: 88779:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 658.776378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.42@tcp [ 658.807739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.683495] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:12:54 (1773684774) [ 660.714250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.721877] LNet: 89362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.723771] LNetError: 89362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.729634] LNet: Removed LNI 192.168.204.42@tcp [ 660.739578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.757363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.765118] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 661.006085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 661.013702] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 661.326339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.42@tcp [ 661.334334] LNetError: 89634:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 661.791334] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 661.791849] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 662.408885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.42@tcp [ 666.393212] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:13:00 (1773684780) [ 666.928104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg442-server [ 667.439406] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:13:01 (1773684781) [ 668.008242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg442-server [ 668.495508] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:13:02 (1773684782) [ 669.046161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg442-server [ 669.543837] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:13:03 (1773684783) [ 669.570070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.578701] LNet: 91259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.580688] LNetError: 91259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.582769] LNet: Removed LNI 192.168.204.42@tcp [ 669.588653] LNet: Removed LNI 192.168.204.42@tcp1 [ 669.597664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.616399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.624032] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 669.857639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 669.880604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 669.896986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 669.920889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 669.937791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 669.951152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 671.850797] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:13:05 (1773684785) [ 671.882839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.892284] LNet: 91984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.894548] LNetError: 91984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.897552] LNetError: 91984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 671.906679] LNet: Removed LNI 192.168.204.42@tcp [ 671.915679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.936072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.945245] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 674.071846] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:13:08 (1773684788) [ 674.099560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.107820] LNet: 92686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.110154] LNetError: 92686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.117533] LNet: Removed LNI 192.168.204.42@tcp [ 674.125890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.145468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.152958] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 676.246984] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:13:10 (1773684790) [ 676.692780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 677.163237] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:13:11 (1773684791) [ 677.296511] LNet: Removed LNI 192.168.204.42@tcp [ 677.495575] Key type .llcrypt unregistered [ 677.496768] Key type ._llcrypt unregistered [ 679.192053] Key type ._llcrypt registered [ 679.193127] Key type .llcrypt registered [ 679.256558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.261893] alg: No test for adler32 (adler32-zlib) [ 680.052959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 680.061576] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 682.649546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 683.991570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.142@tcp [ 684.091608] LNet: 94141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.094539] LNetError: 94141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.107308] LNet: Removed LNI 192.168.204.42@tcp [ 684.237359] Key type .llcrypt unregistered [ 684.238244] Key type ._llcrypt unregistered [ 687.467138] Key type ._llcrypt registered [ 687.468101] Key type .llcrypt registered [ 687.900361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 688.387645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 688.879979] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:13:23 (1773684803) [ 688.999386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.003830] alg: No test for adler32 (adler32-zlib) [ 689.806864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.834807] LNet: 95226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.042315] Key type .llcrypt unregistered [ 690.043544] Key type ._llcrypt unregistered [ 691.903482] Key type ._llcrypt registered [ 691.904398] Key type .llcrypt registered [ 691.960040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 691.966359] alg: No test for adler32 (adler32-zlib) [ 692.773160] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 692.775665] LNet: 95481:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 692.781095] LNet: 95481:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 692.950871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.103117] LNetError: 95493:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.46@tcp has gone from up to down [ 699.107247] LNet: 95539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.109957] LNetError: 95539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.121263] LNet: Removed LNI 192.168.204.42@tcp [ 699.129447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.149943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.160250] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 699.162756] LNet: 95547:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 699.165815] LNet: 95547:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 699.168202] LNetError: 95547:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.44@tcp is a gateway. Can not delete it [ 699.172064] LNetError: 95547:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.204.44@tcp is a router. Can not be deleted [ 699.175426] LNet: 95547:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 701.200528] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:13:35 (1773684815) [ 701.736481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg442-server [ 701.984400] LNetError: 95549:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.47@tcp has gone from up to down [ 701.988541] LNetError: 95549:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 702.240193] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:13:36 (1773684816) [ 702.782739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg442-server [ 703.311896] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:13:37 (1773684817) [ 703.868996] Lustre: Lustre: Build Version: 2.17.50_152_gca7da69 [ 705.055301] LNetError: 95549:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.45@tcp has gone from up to down [ 705.058840] LNetError: 95549:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 705.527155] Key type lgssc registered [ 705.874243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.918650] Key type lgssc unregistered [ 713.058469] LNet: 97974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.060535] LNetError: 97974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.067562] LNet: Removed LNI 192.168.204.42@tcp [ 713.272423] Key type .llcrypt unregistered [ 713.273515] Key type ._llcrypt unregistered [ 717.682124] Key type ._llcrypt registered [ 717.683950] Key type .llcrypt registered [ 718.100523] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:13:52 (1773684832) [ 718.213689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.218954] alg: No test for adler32 (adler32-zlib) [ 719.020050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.043542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 719.060149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 719.074985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 719.089352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 721.042552] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:13:55 (1773684835) [ 721.074656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.082967] LNet: 99373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.091550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.113164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.121719] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 721.413241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.420909] LNetError: 99521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.429252] LNet: Removed LNI 192.168.204.42@tcp [ 721.437575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.456342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 721.464893] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 721.622833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.628835] LNet: 99549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.631107] LNet: 99549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 721.642295] LNet: Removed LNI 192.168.204.42@tcp [ 721.650141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.671968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 721.684173] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 723.744416] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:13:57 (1773684837) [ 723.775138] LNet: 100132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.777443] LNetError: 100132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.779195] LNetError: 100132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 723.785377] LNet: Removed LNI 192.168.204.42@tcp [ 723.994679] Key type .llcrypt unregistered [ 723.996166] Key type ._llcrypt unregistered [ 725.801824] Key type ._llcrypt registered [ 725.802851] Key type .llcrypt registered [ 725.863104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.868579] alg: No test for adler32 (adler32-zlib) [ 726.668815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.695415] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 726.697769] LNet: *** cfs_fail_loc=e003, val=100*** [ 726.939050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 726.948644] Ping reply received with mapping: [ 726.949477] found metadata reply NI[192.168.204.42@tcp] [ 726.950511] 2a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 726.951597] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 726.952684] 5a 5a 5a 5a 5a 5a 5a 5a [ 727.001929] LNet: 100536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.003696] LNetError: 100536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.010309] LNet: Removed LNI 192.168.204.42@tcp [ 727.204402] Key type .llcrypt unregistered [ 727.205514] Key type ._llcrypt unregistered [ 730.508220] Key type ._llcrypt registered [ 730.509930] Key type .llcrypt registered [ 730.994857] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:14:05 (1773684845) [ 731.164273] Key type .llcrypt unregistered [ 731.165439] Key type ._llcrypt unregistered [ 733.163399] Key type ._llcrypt registered [ 733.164941] Key type .llcrypt registered [ 733.233966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.239978] alg: No test for adler32 (adler32-zlib) [ 734.106226] LNetError: 101579:0:(api-ni.c:2734:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 734.237552] Key type .llcrypt unregistered [ 734.239433] Key type ._llcrypt unregistered [ 736.004694] Key type ._llcrypt registered [ 736.005834] Key type .llcrypt registered [ 736.463241] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:14:10 (1773684850) [ 736.645260] Key type .llcrypt unregistered [ 736.646883] Key type ._llcrypt unregistered [ 738.630475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.632488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 740.803083] Key type ._llcrypt registered [ 740.804415] Key type .llcrypt registered [ 740.868027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.874684] alg: No test for adler32 (adler32-zlib) [ 741.677439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.695821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 745.451667] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:14:19 (1773684859) [ 745.487638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.495812] LNet: 103353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.498334] LNetError: 103353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 745.504511] LNet: Removed LNI 10.1.2.3@tcp [ 745.515431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.772438] Key type .llcrypt unregistered [ 745.774130] Key type ._llcrypt unregistered [ 747.725244] Key type ._llcrypt registered [ 747.726683] Key type .llcrypt registered [ 747.787155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.794046] alg: No test for adler32 (adler32-zlib) [ 748.593660] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 748.611553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 748.660615] LNet: 103644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.662959] LNetError: 103644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.671558] LNet: Removed LNI 192.168.204.42@tcp [ 748.887363] Key type .llcrypt unregistered [ 748.888540] Key type ._llcrypt unregistered [ 752.354591] Key type ._llcrypt registered [ 752.355843] Key type .llcrypt registered [ 752.797084] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:14:26 (1773684866) [ 752.991208] Key type .llcrypt unregistered [ 752.992351] Key type ._llcrypt unregistered [ 754.905946] Key type ._llcrypt registered [ 754.907308] Key type .llcrypt registered [ 754.978736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.984359] alg: No test for adler32 (adler32-zlib) [ 757.986414] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:14:32 (1773684872) [ 758.018146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.035636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.065832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.076144] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 758.355879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.370775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 758.388717] LNet: Removed LNI 192.168.204.42@tcp [ 758.476335] LNet: 105385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.627576] Key type .llcrypt unregistered [ 758.628694] Key type ._llcrypt unregistered [ 762.252298] Key type ._llcrypt registered [ 762.253462] Key type .llcrypt registered [ 762.726626] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:14:36 (1773684876) [ 762.843183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.848204] alg: No test for adler32 (adler32-zlib) [ 763.649829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.672124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.681436] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 763.959464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 763.974939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.42@tcp [ 763.984821] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 763.987641] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 763.989514] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 763, 0, 0 [ 763.991455] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 763.992438] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 763.994498] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 763.995828] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 763, 0, 0 [ 763.997988] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 763.999039] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 764.000540] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 764.001749] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 763, 0, 813 [ 764.003595] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 764.004700] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 764.005940] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 764.007128] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 763, 0, 813 [ 764.008849] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 766.074728] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:14:40 (1773684880) [ 766.204668] LNet: 106922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.206882] LNetError: 106922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.216364] LNet: Removed LNI 192.168.204.42@tcp [ 766.421451] Key type .llcrypt unregistered [ 766.423117] Key type ._llcrypt unregistered [ 768.282824] Key type ._llcrypt registered [ 768.284040] Key type .llcrypt registered [ 768.345727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.351100] alg: No test for adler32 (adler32-zlib) [ 769.150819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 769.161591] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 771.847155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 774.322182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.335800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 774.345550] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 774.644849] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.685245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.688158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.690201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 776.821244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 776.828833] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 780.377649] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:14:54 (1773684894) [ 780.410989] LNet: 108433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.414876] LNetError: 108433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.418502] LNet: Removed LNI 192.168.204.42@tcp [ 781.471243] LNet: Removed LNI 192.168.204.42@tcp1 [ 781.473200] LNetError: 108433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 781.476180] LNetError: 108433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 781.484360] LNet: Removed LNI 10.1.2.3@tcp99 [ 781.717446] Key type .llcrypt unregistered [ 781.718651] Key type ._llcrypt unregistered [ 783.775610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.777117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.941488] Key type ._llcrypt registered [ 785.942552] Key type .llcrypt registered [ 786.008307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.014547] alg: No test for adler32 (adler32-zlib) [ 786.826509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.847439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.858822] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 787.104593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.112164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.795515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 787.813093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 787.829958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 787.840248] LNetError: 109182:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 787.853564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 787.871708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 788.021248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 788.046668] LNet: 109202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.049176] LNetError: 109202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.052169] LNet: Removed LNI 192.168.204.42@tcp [ 788.057681] LNet: Removed LNI 10.1.2.3@tcp [ 788.285417] Key type .llcrypt unregistered [ 788.286808] Key type ._llcrypt unregistered [ 791.760834] Key type ._llcrypt registered [ 791.762130] Key type .llcrypt registered [ 792.226217] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:15:06 (1773684906) [ 792.346899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.353985] alg: No test for adler32 (adler32-zlib) [ 793.155876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.176852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 793.185644] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 793.437092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 793.464535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg442-client@tcp [ 795.466305] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:15:09 (1773684909) [ 795.492228] LNet: 110726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.494406] LNetError: 110726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.504384] LNet: Removed LNI 192.168.204.42@tcp [ 795.720418] Key type .llcrypt unregistered [ 795.721724] Key type ._llcrypt unregistered [ 797.607738] Key type ._llcrypt registered [ 797.609184] Key type .llcrypt registered [ 797.675432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.681773] alg: No test for adler32 (adler32-zlib) [ 798.478838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 798.488723] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 798.501425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 800.656421] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:15:14 (1773684914) [ 801.136985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 801.685974] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:15:15 (1773684915) [ 802.185353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 802.717403] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:15:16 (1773684916) [ 802.745778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.754919] LNet: 112143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.758312] LNetError: 112143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.767675] LNet: Removed LNI 192.168.204.42@tcp [ 802.774770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.796463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.804159] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 803.050487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 803.068709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 805.216961] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:15:19 (1773684919) [ 805.251901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.262882] LNet: 112852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.266255] LNetError: 112852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.278070] LNet: Removed LNI 192.168.204.42@tcp [ 805.289603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.313213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 805.328147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 805.558304] Key type .llcrypt unregistered [ 805.559845] Key type ._llcrypt unregistered [ 808.943533] Key type ._llcrypt registered [ 808.944549] Key type .llcrypt registered [ 809.395950] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:15:23 (1773684923) [ 809.515680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.521566] alg: No test for adler32 (adler32-zlib) [ 810.323665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.344698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.353875] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 810.583153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 810.601750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 812.609153] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:15:26 (1773684926) [ 812.638309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.647241] LNet: 114405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.649389] LNetError: 114405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.657595] LNet: Removed LNI 192.168.204.42@tcp [ 812.666709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.686782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 812.698100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 814.709169] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:15:28 (1773684928) [ 814.739295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.748047] LNet: 114985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.757617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.785794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 814.908983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 814.922312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 814.945615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 815.079261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 815.091566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 815.121471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.136330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.159477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 815.297165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 815.312282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 815.335033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 815.468860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 815.482522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 817.518560] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:15:31 (1773684931) [ 817.548997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.559087] LNet: 115618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.561333] LNet: 115618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 817.572788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.598278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.607922] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 817.862811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 817.871433] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 818.173647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 818.197334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 818.276211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.284954] LNetError: 115902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.287483] LNet: Removed LNI 192.168.204.42@tcp [ 818.293319] LNet: Removed LNI 192.168.204.42@tcp2 [ 818.301102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.324232] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 818.327496] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 818.329824] LNet: 115908:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 818.492421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 818.563941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.615134] LNet: Removed LNI 192.168.204.42@tcp [ 819.616452] LNetError: 115945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.619049] LNetError: 115945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 819.627291] LNet: Removed LNI 192.168.204.42@tcp2 [ 819.635766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.656138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.666977] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 819.928684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 819.936916] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 820.228428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 820.242178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 820.261478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 820.332137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.339634] LNet: 116229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.342142] LNet: 116229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 821.407106] LNet: Removed LNI 192.168.204.42@tcp [ 822.437325] LNet: Removed LNI 192.168.204.42@tcp2 [ 822.444337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.465319] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 822.467952] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 822.470076] LNet: 116236:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 822.624332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 822.689655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.696674] LNetError: 116273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.698836] LNetError: 116273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 822.701621] LNet: Removed LNI 192.168.204.42@tcp [ 823.717286] LNet: Removed LNI 192.168.204.42@tcp2 [ 823.724976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.743875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.752023] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 823.980528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 823.987208] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 824.266072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 824.287068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 824.360133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.366499] LNet: 116552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.368877] LNet: 116552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 825.375138] LNet: Removed LNI 192.168.204.42@tcp [ 825.382303] LNet: Removed LNI 192.168.204.42@tcp2 [ 825.390082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.413632] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 825.417082] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 825.419751] LNet: 116558:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 825.579229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 825.645646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.655094] LNet: Removed LNI 192.168.204.42@tcp [ 826.662334] LNet: Removed LNI 192.168.204.42@tcp2 [ 826.671134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.692499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.705244] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 826.959316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 826.968421] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 827.276793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 827.290865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 827.311739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 827.379161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.386879] LNetError: 116879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.388880] LNetError: 116879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 827.391058] LNet: Removed LNI 192.168.204.42@tcp [ 828.453300] LNet: Removed LNI 192.168.204.42@tcp2 [ 828.460334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.480829] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 828.483037] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 828.484704] LNet: 116886:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 828.650804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 828.720314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.727769] LNet: Removed LNI 192.168.204.42@tcp [ 829.797291] LNet: Removed LNI 192.168.204.42@tcp2 [ 829.804634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.824790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.836089] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 830.074254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 830.083617] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 830.367244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 830.379951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 830.401469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 830.518270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.583121] LNet: Removed LNI 192.168.204.42@tcp [ 831.590314] LNet: Removed LNI 192.168.204.42@tcp2 [ 831.597958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.619944] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 831.622923] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 831.773298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 831.885856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.927137] LNet: Removed LNI 192.168.204.42@tcp [ 832.934268] LNet: Removed LNI 192.168.204.42@tcp2 [ 832.941525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.961348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.969943] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 833.210575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 833.219570] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 833.515732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 833.530173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 833.542259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 833.561612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 833.673749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.679730] LNet: 117578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.681428] LNet: 117578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 834.719109] LNet: Removed LNI 192.168.204.42@tcp [ 835.749324] LNet: Removed LNI 192.168.204.42@tcp2 [ 835.757249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.779495] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 835.781734] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 835.783666] LNet: 117585:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 835.786114] LNet: 117585:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 835.948896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 838.158602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 838.676495] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:15:52 (1773684952) [ 838.705624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.715544] LNetError: 118338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.719550] LNetError: 118338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 838.722081] LNet: Removed LNI 192.168.204.42@tcp [ 839.781367] LNet: Removed LNI 192.168.204.42@tcp2 [ 839.790308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.890897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.900573] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 840.150109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.160796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.463772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.472114] LNet: Removed LNI 192.168.204.42@tcp [ 840.478285] LNet: Removed LNI 10.1.2.3@tcp [ 840.485698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.507850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 840.519315] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 840.521031] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.662913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 840.673549] LNet: Removed LNI 192.168.204.42@tcp [ 840.685395] LNet: Removed LNI 10.1.2.3@tcp [ 842.740604] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:15:56 (1773684956) [ 842.776804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.794979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.835818] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.886192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.887962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.890144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.005963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.015350] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 845.255043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 845.262240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.939416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 845.956328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 845.978919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.42@tcp -m PUT -e local_timeout -r 1 [ 846.000897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 846.022885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 846.024449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 846.097218] LNetError: 119280:0:(lib-move.c:4960:lnet_parse()) 192.168.204.42@tcp, src 192.168.204.42@tcp: Dropping PUT (error -108 looking up sender) [ 848.159124] LNet: Removed LNI 192.168.204.42@tcp [ 848.166311] LNet: Removed LNI 10.1.2.3@tcp [ 848.311859] Key type .llcrypt unregistered [ 848.312874] Key type ._llcrypt unregistered [ 851.841452] Key type ._llcrypt registered [ 851.842380] Key type .llcrypt registered [ 852.267691] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:16:06 (1773684966) [ 852.386725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.391391] alg: No test for adler32 (adler32-zlib) [ 853.194040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.313844] LNet: 120590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.522326] Key type .llcrypt unregistered [ 853.523416] Key type ._llcrypt unregistered [ 855.300064] Key type ._llcrypt registered [ 855.300973] Key type .llcrypt registered [ 855.368404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.375709] alg: No test for adler32 (adler32-zlib) [ 856.177062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 856.186363] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 858.803978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 858.859741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 859.050338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 859.129549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 859.142504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 859.155227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 859.166615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 859.179943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 859.192867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 859.205422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 859.217319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 859.224933] LNet: 121038:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 859.233027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 859.244709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 859.257666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 859.267930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 859.279666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 859.293297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 859.490469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 859.562763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 859.572868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 859.584295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 859.596869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 859.609573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 859.621233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 859.632423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 859.643680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 859.654225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 859.665454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 859.676853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 859.687781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 859.698918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 859.710742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 859.911673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 859.987039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 859.998971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 860.010525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 860.021730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 860.034523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 860.046499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 860.058371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 860.070621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 860.076879] LNet: 121272:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 860.079029] LNet: 121272:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 860.086273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 860.097871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 860.109270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 860.123248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 860.135745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 860.148717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.342746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 860.415962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 860.426289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 860.437737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 860.449785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 860.462903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 860.473931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 860.484824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 860.496522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 860.506890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 860.517779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 860.528636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 860.541768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 860.553534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 860.566802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.667478] LNet: 121435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.670297] LNetError: 121435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.678245] LNet: Removed LNI 192.168.204.42@tcp [ 860.806466] Key type .llcrypt unregistered [ 860.807346] Key type ._llcrypt unregistered [ 865.849049] Key type ._llcrypt registered [ 865.850089] Key type .llcrypt registered [ 866.290124] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:16:20 (1773684980) [ 866.403154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.408478] alg: No test for adler32 (adler32-zlib) [ 867.204109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.325525] LNet: 122320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.523309] Key type .llcrypt unregistered [ 867.524514] Key type ._llcrypt unregistered [ 869.388256] Key type ._llcrypt registered [ 869.389734] Key type .llcrypt registered [ 869.460683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 869.466452] alg: No test for adler32 (adler32-zlib) [ 870.270783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 870.280726] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 873.000548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 875.522724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.536732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 875.545891] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 875.855545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 876.050802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 876.126818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.140240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.154105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.166391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.178790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.191680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.203543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.216948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.225635] LNet: 122970:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.234139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 876.248749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 876.262466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 876.275605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.287947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.300190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.504263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 876.580541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.593322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.607060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.620477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.632561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.647217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.661741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.675136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.686559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 876.701144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 876.713898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 876.727186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.740872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.756652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.970348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 877.048890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 877.061720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 877.074136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 877.083914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 877.094981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 877.107988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 877.122319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 877.135907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 877.145642] LNet: 123204:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 877.147537] LNet: 123204:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 877.154515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 877.166385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 877.179198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 877.191220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 877.204763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 877.219790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 877.426510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 877.471348] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.142@tcp, 192.168.204.142@tcp1 [ 877.507552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 877.521525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 877.533952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 877.546832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 877.558933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 877.571439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 877.583628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 877.596093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 877.609478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 877.623266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 877.637614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 877.651833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 877.664850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 877.678044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 879.780713] LNet: 123401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.782995] LNetError: 123401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.785893] LNet: Removed LNI 192.168.204.42@tcp [ 879.793302] LNet: Removed LNI 192.168.204.42@tcp1 [ 879.930545] Key type .llcrypt unregistered [ 879.931434] Key type ._llcrypt unregistered [ 885.041001] Key type ._llcrypt registered [ 885.041882] Key type .llcrypt registered [ 885.467128] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:16:39 (1773684999) [ 887.734039] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:16:41 (1773685001) [ 890.015047] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:16:44 (1773685004) [ 892.306214] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:16:46 (1773685006) [ 894.539886] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:16:48 (1773685008) [ 896.819266] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:16:50 (1773685010) [ 898.950639] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:16:53 (1773685013) [ 901.183413] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:16:55 (1773685015) [ 903.333975] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:16:57 (1773685017) [ 905.408354] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:16:59 (1773685019) [ 907.395848] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:17:01 (1773685021) [ 909.549548] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:17:03 (1773685023) [ 911.681404] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:17:05 (1773685025) [ 913.819978] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:17:07 (1773685027) [ 916.068853] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:17:10 (1773685030) [ 918.262575] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:17:12 (1773685032) [ 920.611312] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:17:14 (1773685034) [ 922.901781] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:17:16 (1773685036) [ 923.022914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 923.029679] alg: No test for adler32 (adler32-zlib) [ 923.832065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.878599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.886695] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 924.279626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.288537] LNet: 134095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.290424] LNetError: 134095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.300340] LNet: Removed LNI 192.168.204.42@tcp [ 924.308487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.329952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 924.340597] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 926.579864] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:17:20 (1773685040) [ 928.848789] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:17:22 (1773685042) [ 931.086751] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:17:25 (1773685045) [ 933.430676] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:17:27 (1773685047) [ 935.599377] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:17:29 (1773685049) [ 937.893131] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:17:31 (1773685051) [ 940.236991] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:17:34 (1773685054) [ 942.446316] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:17:36 (1773685056) [ 944.589164] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:17:38 (1773685058) [ 946.790548] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:17:40 (1773685060) [ 948.979289] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:17:43 (1773685063) [ 951.070355] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:17:45 (1773685065) [ 953.152268] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 843 sec =========== 14:17:47 (1773685067) [ 953.232290] LNet: 141549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.234258] LNetError: 141549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.240738] LNet: Removed LNI 192.168.204.42@tcp [ 953.440377] Key type .llcrypt unregistered [ 953.441233] Key type ._llcrypt unregistered [ 957.038737] Key type ._llcrypt registered [ 957.039824] Key type .llcrypt registered [ 957.210742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.218522] alg: No test for adler32 (adler32-zlib) [ 958.070994] Lustre: Lustre: Build Version: 2.17.50_152_gca7da69 [ 958.147650] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 959.727151] Key type lgssc registered [ 960.095029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.049731] Lustre: Mounted lustre-client [ 987.338794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 994.326848] LustreError: 144000:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff95c1e02df000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 994.340062] Lustre: Unmounted lustre-client [ 1019.790899] Key type lgssc unregistered [ 1019.907444] LNet: 144483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.910780] LNetError: 144483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.920679] LNet: Removed LNI 192.168.204.42@tcp [ 1020.137105] Key type .llcrypt unregistered [ 1020.138655] Key type ._llcrypt unregistered