[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445131489 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006652] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.011014] Yama: becoming mindful. [ 0.011900] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.020223] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025242] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028141] *** VALIDATE tmpfs *** [ 0.030105] *** VALIDATE proc *** [ 0.031290] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038016] Speculative Store Bypass: Vulnerable [ 0.041031] debug: unmapping init [mem 0xffffffffba659000-0xffffffffba660fff] [ 0.044200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052020] ... event mask: 000000070000000f [ 0.053294] rcu: Hierarchical SRCU implementation. [ 0.055546] smp: Bringing up secondary CPUs ... [ 0.057055] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.063049] smp: Brought up 1 node, 4 CPUs [ 0.065025] smpboot: Max logical packages: 1 [ 0.066039] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.287212] node 0 deferred pages initialised in 216ms [ 0.290418] devtmpfs: initialized [ 0.291239] x86/mm: Memory block size: 128MB [ 0.293864] gcov: version magic: 0x41383552 [ 0.294656] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.299091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.302493] pinctrl core: initialized pinctrl subsystem [ 0.304261] [ 0.304721] ************************************************************* [ 0.307015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.310021] ** ** [ 0.312012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.315016] ** ** [ 0.317016] ** This means that this kernel is built to expose internal ** [ 0.320017] ** IOMMU data structures, which may compromise security on ** [ 0.322016] ** your system. ** [ 0.325015] ** ** [ 0.327014] ** If you see this message and you are not debugging the ** [ 0.329012] ** kernel, report this immediately to your vendor! ** [ 0.332016] ** ** [ 0.334010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.336016] ************************************************************* [ 0.338802] NET: Registered protocol family 16 [ 0.341543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.344063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.347069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.351137] cpuidle: using governor menu [ 0.352609] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.356233] PCI: Using configuration type 1 for base access [ 0.358159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.367131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.370023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.374088] cryptd: max_cpu_qlen set to 1000 [ 0.376241] ACPI: Added _OSI(Module Device) [ 0.377025] ACPI: Added _OSI(Processor Device) [ 0.378039] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.380023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.384923] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.391447] ACPI: Interpreter enabled [ 0.392067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.394013] ACPI: Using IOAPIC for interrupt routing [ 0.395143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.398407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.408698] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.411050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.413035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.416088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.420500] acpiphp: Slot [2] registered [ 0.422184] acpiphp: Slot [5] registered [ 0.424220] acpiphp: Slot [6] registered [ 0.426158] acpiphp: Slot [3] registered [ 0.427117] acpiphp: Slot [4] registered [ 0.429110] acpiphp: Slot [7] registered [ 0.430106] acpiphp: Slot [8] registered [ 0.432118] acpiphp: Slot [9] registered [ 0.434106] acpiphp: Slot [10] registered [ 0.435129] acpiphp: Slot [11] registered [ 0.437139] acpiphp: Slot [12] registered [ 0.439112] acpiphp: Slot [13] registered [ 0.440123] acpiphp: Slot [14] registered [ 0.442192] acpiphp: Slot [15] registered [ 0.444104] acpiphp: Slot [16] registered [ 0.445135] acpiphp: Slot [17] registered [ 0.447130] acpiphp: Slot [18] registered [ 0.449115] acpiphp: Slot [19] registered [ 0.450102] acpiphp: Slot [20] registered [ 0.451109] acpiphp: Slot [21] registered [ 0.453108] acpiphp: Slot [22] registered [ 0.454098] acpiphp: Slot [23] registered [ 0.455113] acpiphp: Slot [24] registered [ 0.456142] acpiphp: Slot [25] registered [ 0.458113] acpiphp: Slot [26] registered [ 0.460132] acpiphp: Slot [27] registered [ 0.461094] acpiphp: Slot [28] registered [ 0.462090] acpiphp: Slot [29] registered [ 0.463115] acpiphp: Slot [30] registered [ 0.465188] acpiphp: Slot [31] registered [ 0.467116] PCI host bridge to bus 0000:00 [ 0.468018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.470021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.475030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.479029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.483052] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.485285] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.488088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.491356] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.498016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.502010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.505023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.507018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.509021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.511614] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.514750] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.517057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.520205] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.524016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.535017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.539016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.544452] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.550895] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.556016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.567018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.575112] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.581017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.585017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.604028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.619673] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.622423] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.626504] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.629183] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.636235] iommu: Default domain type: Passthrough [ 0.638677] SCSI subsystem initialized [ 0.641159] ACPI: bus type USB registered [ 0.642157] usbcore: registered new interface driver usbfs [ 0.645102] usbcore: registered new interface driver hub [ 0.647102] usbcore: registered new device driver usb [ 0.649229] pps_core: LinuxPPS API ver. 1 registered [ 0.651025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.655083] PTP clock support registered [ 0.658004] EDAC MC: Ver: 3.0.0 [ 0.660083] PCI: Using ACPI for IRQ routing [ 0.661869] NetLabel: Initializing [ 0.663012] NetLabel: domain hash size = 128 [ 0.665015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.667081] NetLabel: unlabeled traffic allowed by default [ 0.669132] vgaarb: loaded [ 0.671192] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.672019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.680357] clocksource: Switched to clocksource kvm-clock [ 0.785683] VFS: Disk quotas dquot_6.6.0 [ 0.787327] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.791757] *** VALIDATE ramfs *** [ 0.793052] *** VALIDATE hugetlbfs *** [ 0.794190] pnp: PnP ACPI init [ 0.796619] pnp: PnP ACPI: found 6 devices [ 0.825611] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.829538] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.832628] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.835639] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839364] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.843459] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.847549] NET: Registered protocol family 2 [ 0.851072] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857389] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.861993] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867981] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.872140] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875259] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878932] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881504] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884067] NET: Registered protocol family 1 [ 0.886561] RPC: Registered named UNIX socket transport module. [ 0.889017] RPC: Registered udp transport module. [ 0.890806] RPC: Registered tcp transport module. [ 0.892566] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894746] NET: Registered protocol family 44 [ 0.896488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.898960] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901226] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902939] PCI: CLS 0 bytes, default 64 [ 0.904141] Unpacking initramfs... [ 2.341843] debug: unmapping init [mem 0xffff90e9bcc64000-0xffff90e9bffcffff] [ 2.347650] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.350179] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.355298] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.841185] Initialise system trusted keyrings [ 2.842624] Key type blacklist registered [ 2.844563] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.853130] zbud: loaded [ 2.856207] *** VALIDATE nfs *** [ 2.857452] *** VALIDATE nfs4 *** [ 2.858883] pstore: using deflate compression [ 2.862072] Platform Keyring initialized [ 2.968961] NET: Registered protocol family 38 [ 2.970459] Key type asymmetric registered [ 2.971964] Asymmetric key parser 'x509' registered [ 2.974242] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.977940] io scheduler mq-deadline registered [ 2.979252] io scheduler kyber registered [ 2.980487] io scheduler bfq registered [ 2.982544] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.985157] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.987605] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.990019] ACPI: Power Button [PWRF] [ 2.995120] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.002582] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.015540] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.045908] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.077036] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.083578] Non-volatile memory driver v1.3 [ 3.085623] Linux agpgart interface v0.103 [ 3.118549] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.121818] vda: detected capacity change from 0 to 74645504 [ 3.139843] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.143238] vdb: detected capacity change from 0 to 1073741824 [ 3.150219] libphy: Fixed MDIO Bus: probed [ 3.154920] usbcore: registered new interface driver usbserial_generic [ 3.156593] usbserial: USB Serial support registered for generic [ 3.157995] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.161042] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.162541] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.165038] mousedev: PS/2 mouse device common for all mice [ 3.168945] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.173586] rtc_cmos 00:05: RTC can wake from S4 [ 3.177689] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.178056] rtc_cmos 00:05: registered as rtc0 [ 3.183488] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.186543] intel_pstate: CPU model not supported [ 3.188922] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.194990] hid: raw HID events driver (C) Jiri Kosina [ 3.197241] usbcore: registered new interface driver usbhid [ 3.199320] usbhid: USB HID core driver [ 3.201138] drop_monitor: Initializing network drop monitor service [ 3.203723] Initializing XFRM netlink socket [ 3.205935] NET: Registered protocol family 10 [ 3.208930] Segment Routing with IPv6 [ 3.210530] NET: Registered protocol family 17 [ 3.212745] mpls_gso: MPLS GSO support [ 3.218713] RAS: Correctable Errors collector initialized. [ 3.220752] AVX version of gcm_enc/dec engaged. [ 3.222264] AES CTR mode by8 optimization enabled [ 3.312905] sched_clock: Marking stable (3312871814, 0)->(4314758493, -1001886679) [ 3.315958] registered taskstats version 1 [ 3.317775] Loading compiled-in X.509 certificates [ 3.319547] zswap: loaded using pool lzo/zbud [ 3.345471] Key type big_key registered [ 3.360259] Key type encrypted registered [ 3.361720] ima: No TPM chip found, activating TPM-bypass! [ 3.363454] ima: Allocated hash algorithm: sha1 [ 3.364785] ima: No architecture policies found [ 3.366186] evm: Initialising EVM extended attributes: [ 3.367673] evm: security.selinux [ 3.368702] evm: security.ima [ 3.369558] evm: security.capability [ 3.370639] evm: HMAC attrs: 0x1 [ 3.372970] rtc_cmos 00:05: setting system clock to 2026-07-14 16:04:10 UTC (1784045050) [ 3.379066] debug: unmapping init [mem 0xffffffffbb603000-0xffffffffbb7fffff] [ 3.381800] debug: unmapping init [mem 0xffffffffba382000-0xffffffffba658fff] [ 3.390085] Write protecting the kernel read-only data: 28672k [ 3.393257] debug: unmapping init [mem 0xffffffffb8a03000-0xffffffffb8bfffff] [ 3.395622] debug: unmapping init [mem 0xffffffffb9314000-0xffffffffb93fffff] [ 3.427797] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.434893] systemd[1]: Detected virtualization kvm. [ 3.436366] systemd[1]: Detected architecture x86-64. [ 3.437841] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.460236] systemd[1]: No hostname configured. [ 3.461784] systemd[1]: Set hostname to . [ 3.463765] random: systemd: uninitialized urandom read (16 bytes read) [ 3.465757] systemd[1]: Initializing machine ID from random generator. [ 3.495979] random: ln: uninitialized urandom read (6 bytes read) [ 3.600553] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602875] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.607031] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.612250] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.340948] device-mapper: uevent: version 1.0.3 [ 4.343660] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.121302] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initializa[ 5.136958] random: fast init done tion. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.192223] scsi host0: ata_piix [ 5.215485] scsi host1: ata_piix [ 5.217285] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.219783] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.862894] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.875896] random: crng init done [ 9.877589] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.338732] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.554047] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.806561] SELinux: Disabled at runtime. [ 11.868551] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.878516] systemd[1]: Detected virtualization kvm. [ 11.880803] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.403261] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.408180] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.413584] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.420275] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.424762] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.434164] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.444528] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.647669] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.981490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.307986] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.552903] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.599761] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.617681] EDAC sbridge: Ver: 1.1.2 [ 15.153911] Key type dns_resolver registered [ 15.483490] NFS: Registering the id_resolver key type [ 15.485452] Key type id_resolver registered [ 15.486699] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-client login: [ 47.762639] libcfs: loading out-of-tree module taints kernel. [ 47.788973] Key type ._llcrypt registered [ 47.791438] Key type .llcrypt registered [ 48.084760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.091759] alg: No test for adler32 (adler32-zlib) [ 49.133239] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 49.488330] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 51.120158] Key type lgssc registered [ 51.857917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.348018] hrtimer: interrupt took 5059421 ns [ 186.609410] Lustre: Mounted lustre-client [ 191.369743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.350282] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing check_logdir /tmp/testlogs/ [ 210.267485] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing yml_node [ 212.448355] Lustre: lustre-OST0000-osc-ffff90ea0636d000: disconnect after 24s idle [ 214.963558] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 217.858305] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 219.896179] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 221.205699] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jul 14 12:07:47 EDT 2026 [ 237.256688] Lustre: Unmounted lustre-client [ 262.296903] Key type lgssc unregistered [ 262.730297] LNet: 8383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.740648] LNetError: 8383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 262.766742] LNet: Removed LNI 192.168.203.5@tcp [ 263.856079] Key type .llcrypt unregistered [ 263.858716] Key type ._llcrypt unregistered [ 293.003183] Key type ._llcrypt registered [ 293.004405] Key type .llcrypt registered [ 293.181411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 293.199670] alg: No test for adler32 (adler32-zlib) [ 294.231385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 294.329872] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 294.377782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 299.100838] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 299.253425] LNet: 9262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.261735] LNetError: 9262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.285059] LNet: Removed LNI 192.168.203.5@tcp [ 300.132504] Key type .llcrypt unregistered [ 300.134774] Key type ._llcrypt unregistered [ 306.455030] Key type ._llcrypt registered [ 306.460736] Key type .llcrypt registered [ 307.862596] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:09:13 (1784045353) [ 308.243405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 308.257198] alg: No test for adler32 (adler32-zlib) [ 309.194813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.306096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.333465] LNet: 9679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.365795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.441868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 309.472694] LNet: 9686:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 317.288383] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:09:22 (1784045362) [ 317.525445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.563758] LNet: 10263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.585591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.700658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 324.952469] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:09:30 (1784045370) [ 325.185281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.215353] LNet: 10849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.236609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.289306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 331.808792] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:09:37 (1784045377) [ 331.924605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.954994] LNet: 11427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.981861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.089976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 338.077775] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:09:43 (1784045383) [ 338.173157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.214122] LNet: 12012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.240346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.331874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 338.919817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 346.521765] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:09:51 (1784045391) [ 346.663556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.720460] LNet: 12604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.776537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.944488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 347.631463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 348.532942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 356.472348] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:10:02 (1784045402) [ 356.842286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.887246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.958726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 363.610378] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:10:09 (1784045409) [ 363.787696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.817748] LNet: 13790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.827043] LNet: 13790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 363.895057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.980419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 364.024163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 364.558791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 364.591105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 365.148518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 365.207347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 365.891894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 365.937621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 366.333187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 366.379185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 366.919193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 366.954659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 367.444638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 367.481226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 367.973509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 368.032348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 368.645230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 368.695355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 369.337704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 369.410424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 370.124591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 370.176791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 370.731333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 370.772802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.431878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 371.497807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 378.214570] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:10:24 (1784045424) [ 378.445809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.473712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.540480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 378.574910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 385.718811] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:10:31 (1784045431) [ 385.858457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.933811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.065837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 386.132127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.657749] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:10:38 (1784045438) [ 392.785926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.866415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.004891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 393.037503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 398.590798] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:10:44 (1784045444) [ 398.915198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.942250] LNet: 16234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.947225] LNet: 16234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 398.983383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.061585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 399.104480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 404.803415] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:10:50 (1784045450) [ 405.110805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.161528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.231766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 405.294257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 411.981914] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:10:57 (1784045457) [ 412.245692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.289935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.415737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 412.499613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 420.081765] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:11:05 (1784045465) [ 420.180129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.227541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.381675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.872785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.344140] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:11:13 (1784045473) [ 427.425667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.501294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.815925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.113714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.750160] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:11:20 (1784045480) [ 435.049667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.099095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.193695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.379273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.866768] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:11:27 (1784045487) [ 441.963916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.001340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.439879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.679497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 451.903851] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:11:36 (1784045496) [ 452.095036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.171975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.385767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.467125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.210379] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:11:47 (1784045507) [ 463.362760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.401879] LNet: 21062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.409589] LNet: 21062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 463.452486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.561542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.646577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.065530] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:11:56 (1784045516) [ 471.191097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.233552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.365133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 478.152627] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:12:04 (1784045524) [ 478.422240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.451535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.516395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 485.210496] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:12:10 (1784045530) [ 485.352846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.398489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.535457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 486.173409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 493.407673] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:12:18 (1784045538) [ 493.519611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.578267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.674735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 493.733820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 502.711768] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:12:27 (1784045547) [ 502.860158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.936435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.118433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 503.188989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 511.400342] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:12:36 (1784045556) [ 511.535664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.573688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.646980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 511.712816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 519.054342] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:12:44 (1784045564) [ 519.187750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.262738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.411515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 519.465870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 526.766470] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:12:52 (1784045572) [ 527.069676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.108546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.167804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 527.257036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 527.282502] LNetError: 25796:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 527.338234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 527.376138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 534.773589] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:13:00 (1784045580) [ 534.885142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.948407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.041638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7s19qr8bKB [ 535.146359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7s19qr8bKB [ 535.200739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7s19qr8bKB [ 541.957956] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:13:07 (1784045587) [ 542.107708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.177402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.285614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 542.342528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 542.390090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.914205] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:13:15 (1784045595) [ 550.046891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.145209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.289100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.322790] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 552.317624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 554.168541] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 556.178848] Key type lgssc registered [ 557.775550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 564.303205] LNetError: 27557:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.203.5@tcp99 has no available nets [ 568.627356] Key type lgssc unregistered [ 569.250312] LNetError: 28666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.283189] LNet: Removed LNI 192.168.203.5@tcp [ 570.090872] Key type .llcrypt unregistered [ 570.093511] Key type ._llcrypt unregistered [ 578.438855] Key type ._llcrypt registered [ 578.443145] Key type .llcrypt registered [ 580.097287] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:13:45 (1784045625) [ 580.705368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 580.723443] alg: No test for adler32 (adler32-zlib) [ 581.687582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.789115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.691910] LNet: Message forwarding will be enabled in 2s [ 587.808308] LNet: Message forwarding enabled [ 588.303100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 588.349083] LNet: Message forwarding will stop in 2s to 6s [ 591.997186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 592.330050] LNet: Message forwarding disabled [ 594.282810] LNet: Message forwarding will be enabled in 2s [ 597.474992] LNet: Message forwarding enabled [ 601.058289] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:14:06 (1784045646) [ 601.300755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.335725] LNet: 30064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.926759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.015487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 609.801954] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:14:15 (1784045655) [ 610.181925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.209944] LNet: 30645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.253644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.333578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 610.386349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 610.451336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 610.492582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 610.527155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 610.568486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 610.598908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 610.659995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 610.717702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 610.778991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 610.850961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 610.924668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 611.014980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 611.064378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 611.111720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 611.168311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 611.217385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 611.265401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.318568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 611.352187] LNetError: 30689:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 618.461335] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:14:24 (1784045664) [ 618.536174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.559441] LNet: 31253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.586953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.677422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.748279] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:14:31 (1784045671) [ 629.139700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg305-server [ 631.656235] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:14:36 (1784045676) [ 631.950501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.977669] LNet: 32191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.012120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.118488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.157500] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 633.158221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[6-10]@tcp [ 633.183787] LNet: 32325:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 633.233787] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 633.269969] LNetError: 32195:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.8@tcp has gone from up to down [ 634.290137] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 634.302317] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 635.304368] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 635.332708] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 637.361937] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 637.379081] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 640.641395] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:14:46 (1784045686) [ 641.467488] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.7@tcp is being used as a gateway but routing feature is not turned on [ 641.488725] LNetError: 32195:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 642.769151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg305-server [ 644.506376] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:14:50 (1784045690) [ 644.749673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.776028] LNet: 33261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.789496] LNetError: 33261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.829394] LNet: Removed LNI 192.168.203.5@tcp [ 644.855589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.934057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.985089] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 645.767140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 645.813906] LNet: 33379:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 645.829431] LNet: 33379:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 645.857888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 645.862224] LNetError: 33265:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.6@tcp has gone from up to down [ 645.903083] LNetError: 33265:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 652.107254] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:14:57 (1784045697) [ 652.179256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.208154] LNetError: 33965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.255206] LNet: Removed LNI 192.168.203.5@tcp [ 652.286910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.384520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 652.444710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.490837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.586443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.661186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.710267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.790737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 652.907294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.928491] LNet: 34010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.933556] LNet: 34010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 652.951559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.025676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.105931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.151210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.200577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 653.289789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.344915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.416747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.518503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.570553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.653852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 653.806289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.855483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.945127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.067198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.121695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.209823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 654.308441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.369157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.442019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.501527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.570922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.673527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 654.738981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.790172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.904963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.659374] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:15:08 (1784045708) [ 664.696221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg305-server [ 666.764347] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:15:12 (1784045712) [ 669.143333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg305-server [ 671.285868] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:15:16 (1784045716) [ 671.526669] LNet: 35365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.533188] LNet: 35365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 672.390307] Key type .llcrypt unregistered [ 672.403048] Key type ._llcrypt unregistered [ 680.309743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 680.731199] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 680.737862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 680.750116] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 683.509888] Key type ._llcrypt registered [ 683.512186] Key type .llcrypt registered [ 683.982733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 684.010182] alg: No test for adler32 (adler32-zlib) [ 685.010614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.133031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.181909] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 686.337074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 686.378174] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.432659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.460566] LNet: Removed LNI 10.1.2.3@tcp [ 689.661809] LNet: 36095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.673850] LNetError: 36095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.691034] LNet: Removed LNI 192.168.203.5@tcp [ 690.529960] Key type .llcrypt unregistered [ 690.533539] Key type ._llcrypt unregistered [ 703.609978] Key type ._llcrypt registered [ 703.613759] Key type .llcrypt registered [ 705.148107] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:15:50 (1784045750) [ 705.839463] Key type .llcrypt unregistered [ 705.841256] Key type ._llcrypt unregistered [ 712.696351] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.894967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.900844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.915084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.629851] Key type ._llcrypt registered [ 715.639828] Key type .llcrypt registered [ 715.830724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.852107] alg: No test for adler32 (adler32-zlib) [ 716.858118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.955364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.993327] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 717.674570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.692766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.355455] LNet: 37583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.360760] LNetError: 37583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 720.367553] LNet: Removed LNI 10.1.2.3@tcp [ 720.389159] LNet: Removed LNI 192.168.203.5@tcp [ 721.064691] Key type .llcrypt unregistered [ 721.069305] Key type ._llcrypt unregistered [ 733.790850] Key type ._llcrypt registered [ 733.792909] Key type .llcrypt registered [ 734.968333] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:16:20 (1784045780) [ 735.657707] Key type .llcrypt unregistered [ 735.669077] Key type ._llcrypt unregistered [ 742.539311] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 742.669132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 742.671102] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 742.700130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 745.307948] Key type ._llcrypt registered [ 745.313664] Key type .llcrypt registered [ 745.504035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.523644] alg: No test for adler32 (adler32-zlib) [ 746.435922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.512435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 746.531716] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 748.343922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 748.361618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.498901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 750.536648] LNet: Removed LNI 10.1.2.3@tcp [ 750.575184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 750.618458] LNet: Removed LNI 10.1.2.31@tcp [ 750.632544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.673285] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 751.464879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 751.500729] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 752.368378] LNet: 39503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.374788] LNetError: 39503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 752.379328] LNet: Removed LNI 10.1.2.31@tcp [ 752.392271] LNet: Removed LNI 10.1.2.3@tcp [ 753.032499] Key type .llcrypt unregistered [ 753.035883] Key type ._llcrypt unregistered [ 764.967309] Key type ._llcrypt registered [ 764.972787] Key type .llcrypt registered [ 766.450882] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:16:51 (1784045811) [ 766.855583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.879775] alg: No test for adler32 (adler32-zlib) [ 767.842719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.918895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.943329] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 768.939645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.966609] LNet: 40456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.976612] LNetError: 40456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.000512] LNet: Removed LNI 192.168.203.5@tcp [ 769.035155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.089395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.5@tcp [ 769.121692] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 769.920411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.5@tcp [ 769.958904] LNet: Removed LNI 192.168.203.5@tcp [ 770.021221] LNet: 40591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.589961] Key type .llcrypt unregistered [ 770.592831] Key type ._llcrypt unregistered [ 782.609808] Key type ._llcrypt registered [ 782.615292] Key type .llcrypt registered [ 784.206263] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:17:09 (1784045829) [ 785.923680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg305-server [ 787.703367] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:17:13 (1784045833) [ 788.475968] Key type .llcrypt unregistered [ 788.478658] Key type ._llcrypt unregistered [ 794.928612] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.015860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.031186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.039922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 797.715251] Key type ._llcrypt registered [ 797.718086] Key type .llcrypt registered [ 797.897901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 797.910286] alg: No test for adler32 (adler32-zlib) [ 798.828405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.891221] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 799.699979] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 801.722788] LNet: 42429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.734821] LNetError: 42429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.742683] LNet: Removed LNI 192.168.203.5@tcp1 [ 801.755954] LNet: Removed LNI 10.1.2.3@tcp2 [ 802.256413] Key type .llcrypt unregistered [ 802.259645] Key type ._llcrypt unregistered [ 813.472997] Key type ._llcrypt registered [ 813.476229] Key type .llcrypt registered [ 814.565924] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:17:40 (1784045860) [ 814.961509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.984859] alg: No test for adler32 (adler32-zlib) [ 815.894181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.945735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.985898] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 816.796660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.821460] LNet: 43397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.831459] LNetError: 43397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.851670] LNet: Removed LNI 192.168.203.5@tcp [ 816.874454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.930890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 816.951185] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 823.261826] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:17:49 (1784045869) [ 823.539626] LNet: 43988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.549286] LNetError: 43988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.575355] LNet: Removed LNI 192.168.203.5@tcp [ 824.481705] Key type .llcrypt unregistered [ 824.485558] Key type ._llcrypt unregistered [ 831.082255] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 831.265710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 831.270066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 831.274387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 833.849594] Key type ._llcrypt registered [ 833.851564] Key type .llcrypt registered [ 833.998936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.019553] alg: No test for adler32 (adler32-zlib) [ 834.893443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.936451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.954453] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 835.675708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 835.698464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 837.551690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.569604] LNet: 44730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.579898] LNetError: 44730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 837.588541] LNet: Removed LNI 10.1.2.3@tcp [ 837.597800] LNet: Removed LNI 192.168.203.5@tcp [ 837.629906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.734537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 837.783289] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 837.792734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.269820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.295502] LNet: 44757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.301734] LNetError: 44757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 838.309667] LNetError: 44757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 838.320099] LNet: Removed LNI 10.1.2.3@tcp [ 838.340348] LNet: Removed LNI 192.168.203.5@tcp [ 838.384287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.455597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 838.496437] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 838.505715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.126850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 839.155546] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 839.984509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.999393] LNet: 44919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.004688] LNetError: 44919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.009720] LNetError: 44919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.014992] LNet: Removed LNI 10.1.2.3@tcp [ 840.017803] LNet: Removed LNI 192.168.203.5@tcp [ 840.029864] LNet: Removed LNI 192.168.203.5@tcp2 [ 840.055468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.103956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.134376] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 840.146192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.151183] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 846.587247] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:18:12 (1784045892) [ 846.884867] LNet: 45516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.905649] LNetError: 45516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 846.918680] LNetError: 45516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 846.927758] LNet: Removed LNI 10.1.2.3@tcp [ 846.934421] LNet: Removed LNI 192.168.203.5@tcp [ 846.943593] LNet: Removed LNI 192.168.203.5@tcp2 [ 847.659742] Key type .llcrypt unregistered [ 847.663633] Key type ._llcrypt unregistered [ 854.858733] Key type ._llcrypt registered [ 854.860404] Key type .llcrypt registered [ 855.061319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.080213] alg: No test for adler32 (adler32-zlib) [ 856.183954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.320765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.405933] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 857.914689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.968542] LNet: 45911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.982698] LNetError: 45911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.997967] LNet: Removed LNI 192.168.203.5@tcp [ 858.024794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.137886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 858.170456] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 858.795028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.817922] LNet: 45941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.823766] LNetError: 45941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.842884] LNet: Removed LNI 192.168.203.5@tcp [ 858.866289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.971615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.014041] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 859.890514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.921816] LNet: 45971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.929840] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.953293] LNet: Removed LNI 192.168.203.5@tcp [ 860.005874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.124781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.175006] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 861.054360] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.394783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.407033] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.414283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.059935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.134345] LNet: 46051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.159546] LNetError: 46051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.199463] LNet: Removed LNI 192.168.203.5@tcp [ 864.268404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.413618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.437787] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 872.950550] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:18:38 (1784045918) [ 873.105893] LNet: 46641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.115606] LNetError: 46641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.135820] LNet: Removed LNI 192.168.203.5@tcp [ 874.058702] Key type .llcrypt unregistered [ 874.071095] Key type ._llcrypt unregistered [ 881.052597] Key type ._llcrypt registered [ 881.056981] Key type .llcrypt registered [ 881.390052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.420608] alg: No test for adler32 (adler32-zlib) [ 882.450427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.521735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.565520] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 883.562914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.598095] LNet: 47046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.605831] LNetError: 47046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.629691] LNet: Removed LNI 192.168.203.5@tcp [ 883.658203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.745244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 883.771956] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 890.347509] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:18:56 (1784045936) [ 890.531235] LNet: 47638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.538560] LNetError: 47638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.554187] LNet: Removed LNI 192.168.203.5@tcp [ 891.122553] Key type .llcrypt unregistered [ 891.124926] Key type ._llcrypt unregistered [ 897.395862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.515407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.523144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.526911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 899.965606] Key type ._llcrypt registered [ 899.967916] Key type .llcrypt registered [ 900.248891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.274438] alg: No test for adler32 (adler32-zlib) [ 901.244955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.317046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 901.338389] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 902.229713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.255558] LNet: 48087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.264564] LNetError: 48087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.274817] LNet: Removed LNI 192.168.203.5@tcp [ 902.315371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.377993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 902.396264] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 908.863310] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:19:14 (1784045954) [ 908.989420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.017972] LNet: 48682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.030489] LNetError: 48682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.044778] LNet: Removed LNI 192.168.203.5@tcp [ 909.076086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.211911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 914.938842] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:19:20 (1784045960) [ 915.186624] LNet: 49270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.806560] Key type .llcrypt unregistered [ 915.809165] Key type ._llcrypt unregistered [ 922.947183] Key type ._llcrypt registered [ 922.951902] Key type .llcrypt registered [ 923.227888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.253054] alg: No test for adler32 (adler32-zlib) [ 924.246866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.359300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.399973] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 925.271440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.288428] LNet: 49667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.294444] LNetError: 49667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.339089] LNet: Removed LNI 192.168.203.5@tcp [ 925.386257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.450485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 925.480576] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 932.493774] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:19:38 (1784045978) [ 932.731424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.785149] LNet: 50260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.792862] LNetError: 50260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.810431] LNet: Removed LNI 192.168.203.5@tcp [ 932.826394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.882648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 939.612213] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:19:45 (1784045985) [ 941.372168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 943.487401] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:19:48 (1784045988) [ 945.189708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 946.969853] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:19:52 (1784045992) [ 948.418870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 950.485974] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:19:55 (1784045995) [ 951.934087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 953.936203] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:19:59 (1784045999) [ 955.190852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 956.966925] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:20:02 (1784046002) [ 957.064362] LNet: 52306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.969664] Key type .llcrypt unregistered [ 957.977326] Key type ._llcrypt unregistered [ 965.244545] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.436312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.453298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.467027] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.973795] Key type ._llcrypt registered [ 967.976437] Key type .llcrypt registered [ 968.144447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.160533] alg: No test for adler32 (adler32-zlib) [ 969.118792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.282880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 969.334236] LNet: Added LNI 192.168.203.5@tcp [8/512/0/180] [ 969.344622] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 969.346858] LNet: 52634:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 969.475877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.504492] LNet: 52653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.516762] LNetError: 52653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 969.525514] LNet: Removed LNI 10.1.2.3@tcp [ 969.528529] LNetError: 52653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.571067] LNet: Removed LNI 192.168.203.5@tcp [ 969.597732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.679630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 969.722450] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 969.812559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.834886] LNet: 52678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.851060] LNetError: 52678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.877261] LNet: Removed LNI 192.168.203.5@tcp [ 969.932684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.031617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.068565] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 970.075970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.078158] LNet: 52686:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 970.159301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.178064] LNet: 52704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.183493] LNetError: 52704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.188843] LNet: Removed LNI 10.1.2.3@tcp [ 970.192551] LNetError: 52704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.201166] LNet: Removed LNI 192.168.203.5@tcp [ 970.219126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.279801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.312705] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 970.319924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.322708] LNet: 52712:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 970.531432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.564400] LNet: 52738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.570512] LNetError: 52738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.577647] LNet: Removed LNI 10.1.2.3@tcp [ 970.580851] LNetError: 52738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.593580] LNet: Removed LNI 192.168.203.5@tcp [ 970.610463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.660695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.679463] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 970.684741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 970.778578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.822726] LNet: 52766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.830643] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.845530] LNet: Removed LNI 10.1.2.3@tcp [ 970.849625] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.868761] LNet: Removed LNI 192.168.203.5@tcp [ 970.892554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.942915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.963478] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 970.966979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 970.969250] LNet: 52774:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 971.030314] LNet: Removed LNI 10.1.2.3@tcp [ 971.036591] LNet: Removed LNI 192.168.203.5@tcp [ 971.638953] Key type .llcrypt unregistered [ 971.644927] Key type ._llcrypt unregistered [ 982.835966] Key type ._llcrypt registered [ 982.837965] Key type .llcrypt registered [ 983.936596] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:20:29 (1784046029) [ 984.436052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 984.459544] alg: No test for adler32 (adler32-zlib) [ 985.405156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.449935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.494903] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 986.412848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.5 [ 986.451263] LNet: 53750:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 986.472454] LNet: Removed LNI 192.168.203.5@tcp [ 993.142348] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:20:38 (1784046038) [ 993.381764] LNet: 54322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.391381] LNetError: 54322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.403615] LNet: Removed LNI 192.168.203.5@tcp [ 994.022404] Key type .llcrypt unregistered [ 994.024603] Key type ._llcrypt unregistered [ 1000.480237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.700882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1003.452796] Key type ._llcrypt registered [ 1003.455503] Key type .llcrypt registered [ 1003.622103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.639146] alg: No test for adler32 (adler32-zlib) [ 1004.922492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1010.371766] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:20:56 (1784046056) [ 1010.571704] LNet: 55235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.578308] LNetError: 55235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.595848] LNet: Removed LNI 10.1.2.3@tcp [ 1011.155427] Key type .llcrypt unregistered [ 1011.158691] Key type ._llcrypt unregistered [ 1016.810105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1017.044454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1017.054159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1017.062931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1019.777308] Key type ._llcrypt registered [ 1019.787364] Key type .llcrypt registered [ 1019.988161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.015617] alg: No test for adler32 (adler32-zlib) [ 1020.899780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1026.539379] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:21:12 (1784046072) [ 1028.070399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1029.589897] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:21:15 (1784046075) [ 1029.837517] LNet: 56421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.846595] LNetError: 56421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1029.860787] LNet: Removed LNI 10.1.2.3@tcp [ 1030.484554] Key type .llcrypt unregistered [ 1030.489577] Key type ._llcrypt unregistered [ 1036.708398] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1036.945887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1036.950950] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1036.964400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1038.523255] Key type ._llcrypt registered [ 1038.527373] Key type .llcrypt registered [ 1038.704457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.717443] alg: No test for adler32 (adler32-zlib) [ 1039.729666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.791837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.140775] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:21:33 (1784046093) [ 1048.543228] LNet: 57644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.551989] LNetError: 57644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.569859] LNet: Removed LNI 10.1.2.3@tcp [ 1049.219699] Key type .llcrypt unregistered [ 1049.223687] Key type ._llcrypt unregistered [ 1054.931670] Key type ._llcrypt registered [ 1054.933355] Key type .llcrypt registered [ 1055.171759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.198877] alg: No test for adler32 (adler32-zlib) [ 1056.103608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1056.147666] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1063.320977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1064.019080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 1064.067670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 1064.110454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1064.151566] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1064.352806] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1064.429520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.477357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.512745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.540376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.576472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.609561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.651944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.683472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.706231] LNet: 58081:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.724979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.754678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.787071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.823981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.868274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.904256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.940373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.442479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 1065.495288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 1065.525423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1065.549702] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1065.764920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.804233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.860418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.903410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.950044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.984145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.020659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.045848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.074634] LNet: 58190:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.094228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.133993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.185850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.219983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.253807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.301930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.348773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.797216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 1066.835056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 1066.891248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1066.910978] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1067.113875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.165411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.208601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.232881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.258207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.286594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.323203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.364820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.389938] LNet: 58300:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.413059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.448246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.486104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.508789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.534907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.585102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.626795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.102412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 1068.140282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 1068.173141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1068.385673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.421140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.457798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.486418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.527459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.570113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.608160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.653973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.682652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.711784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.740456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.771788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.808136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.855854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.893703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1069.465502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 1069.499649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 1069.556807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1069.588031] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1069.597484] LNet: Skipped 2 previous similar messages [ 1069.782917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.814261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.884397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.953548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.009930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.067650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.114537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.157914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.182242] LNet: 58518:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.189823] LNet: 58518:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1070.216783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.265724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.300879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.333412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.366681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.409799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.445541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1070.998547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 1071.025320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 1071.067045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1071.300275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.339405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.386088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.419860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.457514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.491915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.519478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.549860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.583993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.621072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.644738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.673828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.697990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.723432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.752813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1071.961817] LNet: 58668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.967878] LNetError: 58668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.992791] LNet: Removed LNI 192.168.203.5@tcp [ 1072.450937] Key type .llcrypt unregistered [ 1072.460426] Key type ._llcrypt unregistered [ 1091.707887] Key type ._llcrypt registered [ 1091.710986] Key type .llcrypt registered [ 1093.229980] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:22:18 (1784046138) [ 1094.300929] Key type .llcrypt unregistered [ 1094.303121] Key type ._llcrypt unregistered [ 1100.507462] Key type ._llcrypt registered [ 1100.510873] Key type .llcrypt registered [ 1100.669986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.688349] alg: No test for adler32 (adler32-zlib) [ 1101.597463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.629856] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1109.107486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1120.153624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1120.186876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1120.216223] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1121.784463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 1121.878480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 1121.959252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 1122.030207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 1122.081839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 1122.125206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 1122.170528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 1122.205576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 1122.247188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1122.263050] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1122.267766] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1122.497561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.537303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.569748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.603930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.645880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.686848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.731746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.773589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.792788] LNet: 60137:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.817417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.867288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.910063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.956831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.002784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.042453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.086798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.655318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 1123.687530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 1123.737349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 1123.810621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 1123.883444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 1123.953227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 1124.018195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 1124.087833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 1124.145517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1124.165528] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1124.178984] LNet: Skipped 2 previous similar messages [ 1124.522317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.559625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.619737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.658724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.709654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.779994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.836340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.869587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.892626] LNet: 60259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.925818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.972081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.012991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.046824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.082638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.124957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.186802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.829696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 1125.882978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 1125.929676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 1125.971288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 1126.022785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 1126.073108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 1126.131656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 1126.194498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 1126.270421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1126.305862] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1126.310849] LNet: Skipped 2 previous similar messages [ 1126.590360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.644627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.699781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.750620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.815231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.851582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.883713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.909980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.927778] LNet: 60381:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.957278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.019596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.059607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.102320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.148246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.177713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.243523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.870129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 1127.914877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 1127.951701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 1127.998862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 1128.044587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 1128.096735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 1128.126905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 1128.190854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 1128.255123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1128.521443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.550742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.596953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.643530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.698753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.752966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.792489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.821122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.865209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.904304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.941715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.969469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.020212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.067716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.108865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.554650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 1129.589577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 1129.647778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 1129.694554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 1129.749498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 1129.799977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 1129.863559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 1129.926692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 1129.995739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1130.021684] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1130.029550] LNet: Skipped 5 previous similar messages [ 1130.295805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.328987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.366254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.397906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.434968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.502162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.539563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.573652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.614972] LNet: 60624:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.628800] LNet: 60624:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1130.660957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.704251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.735736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.788580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.823718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.859679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.904558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1131.460712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 1131.500237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 1131.543278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 1131.587933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 1131.642659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 1131.717839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 1131.749378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 1131.792668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 1131.845797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1132.121529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.149658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.180740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.218960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.260461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.294993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.334403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.375812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.408926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.437915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.470594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.504157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.536479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.568557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.595578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.633070] LNet: 60821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.648230] LNetError: 60821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.658791] LNet: Removed LNI 192.168.203.5@tcp [ 1139.688374] LNet: Removed LNI 192.168.203.5@tcp1 [ 1140.237443] Key type .llcrypt unregistered [ 1140.241374] Key type ._llcrypt unregistered [ 1155.219787] Key type ._llcrypt registered [ 1155.222687] Key type .llcrypt registered [ 1157.200847] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:23:22 (1784046202) [ 1158.115467] Key type .llcrypt unregistered [ 1158.122072] Key type ._llcrypt unregistered [ 1165.034434] Key type ._llcrypt registered [ 1165.039235] Key type .llcrypt registered [ 1165.282119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1165.300966] alg: No test for adler32 (adler32-zlib) [ 1166.191382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1166.227093] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1172.971483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1173.668184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1173.724194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1173.775512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1173.791796] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1173.963837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.991594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.016871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.042662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.068956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.093649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.116867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.151685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.179325] LNet: 62072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.200755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.238267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.278971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.310772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.356315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.407674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.438407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.972305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1175.023071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1175.064535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1175.114898] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1175.285840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.310654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.333433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.366463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.410356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.461706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.501841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.548156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.574641] LNet: 62182:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.602591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.642633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.677953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.713981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.754373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.800534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.848332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.380116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1176.433837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1176.484723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1176.506374] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1176.768631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.813374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.852702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.897964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.926674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.946735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.976279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.028621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.041946] LNet: 62291:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.057147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.077689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.107964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.135729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.164664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.198845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.254783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.499339] LNet: 62332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.512190] LNetError: 62332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.544651] LNet: Removed LNI 192.168.203.5@tcp [ 1177.854946] Key type .llcrypt unregistered [ 1177.859615] Key type ._llcrypt unregistered [ 1193.533576] Key type ._llcrypt registered [ 1193.538761] Key type .llcrypt registered [ 1194.873858] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:24:00 (1784046240) [ 1195.622707] Key type .llcrypt unregistered [ 1195.628514] Key type ._llcrypt unregistered [ 1201.272148] Key type ._llcrypt registered [ 1201.274577] Key type .llcrypt registered [ 1201.464701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.486169] alg: No test for adler32 (adler32-zlib) [ 1202.413032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.462302] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1209.178976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1217.289890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.342482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1217.372290] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1218.703249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1218.752178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 1218.811529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1218.858918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 1218.918930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1218.971081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 1219.028657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1219.086526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 1219.129086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1219.156613] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1219.165278] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1219.367404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.404890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.431486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.453555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.474701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.499383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.525948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.555469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.570278] LNet: 63799:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.592461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.628504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.662129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.694650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.730744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.783156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.808212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1219.843519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.397432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1220.456206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 1220.500389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1220.565247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 1220.609644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1220.663544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 1220.713157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1220.746115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 1220.774600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1220.792329] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1220.798599] LNet: Skipped 2 previous similar messages [ 1220.977180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.000719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.030829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.069996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.099882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.132604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.171399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.200411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.211959] LNet: 63923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.232272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.277641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.309408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.336693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.367177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.394668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.428565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1221.462820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.891445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1221.925062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 1221.963888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1221.996808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 1222.029721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1222.065427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 1222.096855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1222.134346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 1222.169616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1222.208323] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1222.213906] LNet: Skipped 1 previous similar message [ 1222.374202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.410505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.443975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.486426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.510957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.553669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.591581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.616188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.629961] LNet: 64046:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.650387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.678835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.719969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.772230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.799680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.832932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.860480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1222.907638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.605813] LNet: 64124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.611775] LNetError: 64124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.616864] LNet: Removed LNI 192.168.203.5@tcp [ 1226.628986] LNet: Removed LNI 192.168.203.5@tcp1 [ 1227.037697] Key type .llcrypt unregistered [ 1227.043061] Key type ._llcrypt unregistered [ 1243.384332] Key type ._llcrypt registered [ 1243.386772] Key type .llcrypt registered [ 1244.897618] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:24:50 (1784046290) [ 1246.630456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1249.137849] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:24:54 (1784046294) [ 1250.326754] Key type .llcrypt unregistered [ 1250.328912] Key type ._llcrypt unregistered [ 1258.658616] Key type ._llcrypt registered [ 1258.661132] Key type .llcrypt registered [ 1259.006704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1259.054399] alg: No test for adler32 (adler32-zlib) [ 1260.150609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1260.210662] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1268.402681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1269.320026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1269.381157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1269.470250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1269.504060] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1269.750675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.803505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.869312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.985468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.058453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.131924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.200400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.266645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.298371] LNet: 65669:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.331488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.420520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.476799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.496488] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1270.505762] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1270.517076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.594110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.673779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.977745] LNet: 65708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.985614] LNetError: 65708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.010581] LNet: Removed LNI 192.168.203.5@tcp [ 1271.696455] Key type .llcrypt unregistered [ 1271.704778] Key type ._llcrypt unregistered [ 1294.607872] Key type ._llcrypt registered [ 1294.609940] Key type .llcrypt registered [ 1294.813871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.829861] alg: No test for adler32 (adler32-zlib) [ 1295.722463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.776203] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1303.089987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1311.982749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.013336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.051933] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1313.678504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1313.702729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 1313.734428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1313.780306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 1313.823133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1313.861292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 1313.904923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1313.938902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 1313.983494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1314.001928] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1314.173758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.206782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1314.208107] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1314.219059] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1314.240770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.270609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1314.313277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1314.349703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1314.380902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1314.426949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1314.444745] LNet: 66589:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1314.467191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1314.493877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1314.550099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.596697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1314.643798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1314.694425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.988724] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1319.345665] LNet: 66663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.351743] LNetError: 66663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.364967] LNet: Removed LNI 192.168.203.5@tcp [ 1319.383975] LNet: Removed LNI 192.168.203.5@tcp1 [ 1319.798669] Key type .llcrypt unregistered [ 1319.800413] Key type ._llcrypt unregistered [ 1334.296945] Key type ._llcrypt registered [ 1334.299536] Key type .llcrypt registered [ 1335.544960] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:26:21 (1784046381) [ 1335.915677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.935315] alg: No test for adler32 (adler32-zlib) [ 1336.836863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.884110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.909882] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1337.572174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.594709] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1338.345504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1338.401615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1338.440987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1338.474194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1338.507562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1338.539616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1338.570249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1338.585346] LNetError: 67764:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1338.912517] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1338.917775] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1340.960591] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1342.079358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1344.033869] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1344.613824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1344.660540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.725425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1344.741566] LNet: 67862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.746692] LNetError: 67862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.750847] LNet: Removed LNI 192.168.203.5@tcp [ 1344.761290] LNet: Removed LNI 192.168.203.5@tcp1 [ 1344.778621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.849988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1344.883487] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1345.722723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1345.744934] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1346.522854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1346.572617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.642770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1346.688103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1346.726599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1346.774499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1346.824523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1346.879127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1346.898761] LNetError: 68134:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1347.874456] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1350.533793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1354.018520] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1354.025742] LNet: Skipped 1 previous similar message [ 1358.445764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1358.475438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.503861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.044581] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:26:50 (1784046410) [ 1365.225528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.256961] LNet: 68851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.263406] LNetError: 68851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.268441] LNetError: 68851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1365.274436] LNet: Removed LNI 192.168.203.5@tcp [ 1365.293394] LNet: Removed LNI 192.168.203.5@tcp1 [ 1365.308613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.361089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.389194] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1365.982418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.007145] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1366.619621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1366.695673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1366.742233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1366.768928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1366.822744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1366.863399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.915489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 1367.022603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1367.392368] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.5@tcp [ 1367.392475] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1367.406073] LNet: Skipped 1 previous similar message [ 1382.491214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.607677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1382.638624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.663037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 500 [ 1382.747507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.404120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.494842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.534615] LNet: 69344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.547121] LNetError: 69344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.560349] LNetError: 69344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1389.571940] LNet: Removed LNI 192.168.203.5@tcp [ 1389.584787] LNet: Removed LNI 192.168.203.5@tcp1 [ 1389.619630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.689093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.723514] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1390.557945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.575393] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1391.485952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1391.549948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.628653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1391.665647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1391.700593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1391.731568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1391.765430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1391.803807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 1392.736827] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1392.751185] LNet: Skipped 3 previous similar messages [ 1393.235479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.752806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.791479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1403.823631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.907496] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:27:36 (1784046456) [ 1411.336790] LNet: 70389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.364650] LNetError: 70389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.376803] LNetError: 70389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.387664] LNet: Removed LNI 192.168.203.5@tcp [ 1411.404903] LNet: Removed LNI 192.168.203.5@tcp1 [ 1412.090681] Key type .llcrypt unregistered [ 1412.094898] Key type ._llcrypt unregistered [ 1418.899582] Key type ._llcrypt registered [ 1418.901381] Key type .llcrypt registered [ 1419.106611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.116428] alg: No test for adler32 (adler32-zlib) [ 1420.008544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1420.037296] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1429.057619] Lustre: DEBUG MARKER: Initial discovery [ 1429.163989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1431.401236] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1431.520097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1431.599100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1431.625844] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1431.646061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1431.667524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -l 3 [ 1431.707804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -e local_error [ 1431.769222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m PUT -l 6 [ 1432.355652] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1433.207746] Lustre: DEBUG MARKER: Force oleg305-client.virtnet to discover 192.168.203.105@tcp (in background) [ 1433.283478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1434.585342] Lustre: DEBUG MARKER: Force oleg305-server to discover 192.168.203.5@tcp [ 1436.321700] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1436.974765] Lustre: DEBUG MARKER: Wait for 71145 [ 1438.583501] Lustre: DEBUG MARKER: Finished wait on 71145 [ 1438.646756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1438.693655] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1438.735783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1439.039840] LNet: 71602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.048650] LNetError: 71602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.068042] LNet: Removed LNI 192.168.203.5@tcp [ 1439.614547] Key type .llcrypt unregistered [ 1439.617852] Key type ._llcrypt unregistered [ 1454.467672] Key type ._llcrypt registered [ 1454.470386] Key type .llcrypt registered [ 1455.710135] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:28:21 (1784046501) [ 1456.235586] Key type .llcrypt unregistered [ 1456.237344] Key type ._llcrypt unregistered [ 1461.181816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1461.264891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1461.269232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1461.275303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1463.784386] Key type ._llcrypt registered [ 1463.789074] Key type .llcrypt registered [ 1463.941396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.951266] alg: No test for adler32 (adler32-zlib) [ 1464.836176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.891788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.917288] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1465.595654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1465.623813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1473.006507] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:28:38 (1784046518) [ 1473.151378] LNet: 73689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.162992] LNetError: 73689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1473.170737] LNet: Removed LNI 10.1.2.3@tcp [ 1473.189356] LNet: Removed LNI 192.168.203.5@tcp [ 1473.799752] Key type .llcrypt unregistered [ 1473.803839] Key type ._llcrypt unregistered [ 1479.889932] Key type ._llcrypt registered [ 1479.893416] Key type .llcrypt registered [ 1480.099380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1480.117243] alg: No test for adler32 (adler32-zlib) [ 1481.064511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.141429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.180866] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1481.942658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1481.963635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1482.916344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.5@tcp [ 1484.000209] LNetError: Unexpected error -22 connecting to 192.168.203.5@tcp at host 192.168.203.5:988 [ 1489.537639] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:28:55 (1784046535) [ 1489.619690] LNet: 74830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.631424] LNetError: 74830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1489.640101] LNet: Removed LNI 10.1.2.3@tcp [ 1489.656983] LNet: Removed LNI 192.168.203.5@tcp [ 1490.350575] Key type .llcrypt unregistered [ 1490.352306] Key type ._llcrypt unregistered [ 1495.868587] Key type ._llcrypt registered [ 1495.874927] Key type .llcrypt registered [ 1496.094833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.109638] alg: No test for adler32 (adler32-zlib) [ 1497.004904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.071400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.117412] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1497.809671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1497.832598] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1498.645981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.5@tcp1 --nid 192.168.203.5@tcp2 [ 1502.559056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp2 [ 1502.583500] LNetError: 75675:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.5@tcp1 to 192.168.203.5@tcp2 - Route table may be misconfigured [ 1502.618959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp2 192.168.203.5@tcp1 [ 1508.350318] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:29:14 (1784046554) [ 1508.477838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.529365] LNet: 76243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.540904] LNetError: 76243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.557791] LNet: Removed LNI 192.168.203.5@tcp1 [ 1509.608054] LNet: Removed LNI 192.168.203.5@tcp2 [ 1509.659876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.741957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.773036] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1510.382520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.398825] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1511.147228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1511.213916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.5@tcp [ 1511.278202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp1 [ 1511.340219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp -e network_timeout [ 1511.400416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -e network_timeout [ 1511.475248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -e network_timeout [ 1511.547896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -e network_timeout [ 1511.601485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1511.617376] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1511.632523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1511.730581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.779159] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1511.809764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1518.022090] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:29:23 (1784046563) [ 1518.083340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.104525] LNet: 77101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.115266] LNetError: 77101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.122207] LNetError: 77101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1518.128535] LNet: Removed LNI 192.168.203.5@tcp [ 1519.147483] LNet: Removed LNI 192.168.203.5@tcp1 [ 1519.175991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.259396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1519.529213] LNet: 77147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.863685] Key type .llcrypt unregistered [ 1519.866733] Key type ._llcrypt unregistered [ 1529.795190] Key type ._llcrypt registered [ 1529.800564] Key type .llcrypt registered [ 1530.937880] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:29:36 (1784046576) [ 1531.315128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.336126] alg: No test for adler32 (adler32-zlib) [ 1532.261963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.333905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1534.861323] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:29:40 (1784046580) [ 1535.003864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.030472] LNet: 78245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.052762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.099838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.125535] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1535.743141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1535.758756] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1536.531172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1536.583578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp1 [ 1536.631681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp1 [ 1542.213106] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:29:47 (1784046587) [ 1543.548160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg305-server [ 1544.773460] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:29:50 (1784046590) [ 1546.032760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg305-server [ 1547.817829] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:29:53 (1784046593) [ 1549.347601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg305-server [ 1550.818780] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:29:56 (1784046596) [ 1552.610302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg305-server [ 1554.230788] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:30:00 (1784046600) [ 1555.502623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg305-server [ 1557.186851] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:30:02 (1784046602) [ 1558.825884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg305-server [ 1560.415895] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:30:06 (1784046606) [ 1561.909953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg305-server [ 1563.365723] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:30:09 (1784046609) [ 1564.829827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg305-server [ 1566.155396] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:30:12 (1784046612) [ 1567.594613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg305-server [ 1569.165504] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:30:14 (1784046614) [ 1569.245725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.261667] LNet: 82115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.268249] LNetError: 82115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.276928] LNet: Removed LNI 192.168.203.5@tcp [ 1569.292701] LNet: Removed LNI 192.168.203.5@tcp1 [ 1569.314233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.370122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.399373] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1570.056973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1570.134232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1570.287180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.314361] LNet: 82268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.322718] LNetError: 82268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.333481] LNetError: 82268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.347578] LNet: Removed LNI 192.168.203.5@tcp [ 1570.373701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.441535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1570.461859] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1571.054443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1571.208118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.265117] LNet: Removed LNI 192.168.203.5@tcp [ 1572.293881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.362861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.393977] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1573.051660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1573.116328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1573.294622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.308936] LNet: 82561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.313368] LNet: 82561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1573.320395] LNetError: 82561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.328247] LNetError: 82561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1573.341152] LNet: Removed LNI 192.168.203.5@tcp [ 1573.362864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.438591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1573.474676] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1574.100513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1574.271123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.334755] LNet: Removed LNI 192.168.203.5@tcp [ 1575.365760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.425936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.447895] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1576.148363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1576.230779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1576.445362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.465985] LNetError: 82853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.469924] LNetError: 82853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1576.490349] LNet: Removed LNI 192.168.203.5@tcp [ 1576.512712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.582916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1576.609894] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1577.317399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1577.511227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.537886] LNet: 82998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.547064] LNet: 82998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1578.599072] LNet: Removed LNI 192.168.203.5@tcp [ 1578.624972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.679977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.696073] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1579.138282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1579.205316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1579.400486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.457785] LNet: Removed LNI 192.168.203.5@tcp [ 1580.489655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.535142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1580.570316] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1581.162949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1582.592734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.620712] LNetError: 83300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.628130] LNetError: 83300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1582.665612] LNet: Removed LNI 192.168.203.5@tcp [ 1582.688789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.756664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.781274] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1583.453332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1583.519872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1584.855588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.963748] LNet: Removed LNI 192.168.203.5@tcp [ 1585.983296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.051663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1586.096294] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1586.775640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1587.018242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.036473] LNet: 83602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.048146] LNet: 83602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1588.073823] LNet: Removed LNI 192.168.203.5@tcp [ 1588.111600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.197825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.229732] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1589.155836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1589.253986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1589.532894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.568319] LNet: Removed LNI 192.168.203.5@tcp [ 1590.604826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.684866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1590.709978] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1591.534475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1591.842175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.868338] LNetError: 83895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.878583] LNetError: 83895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1591.899494] LNet: Removed LNI 192.168.203.5@tcp [ 1591.936903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.012040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.049086] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1592.871584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1592.938696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.965877] LNet: Removed LNI 192.168.203.5@tcp [ 1592.988048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.058743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.093028] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1593.786583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1593.861468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.901262] LNet: Removed LNI 192.168.203.5@tcp [ 1593.924659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.982754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1594.055499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.082621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.133633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1594.209188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.240621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.286720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1594.354413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.384504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.444849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1594.472564] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1595.079457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.126824] LNet: Removed LNI 192.168.203.5@tcp [ 1595.148891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.236686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1601.028381] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:30:47 (1784046647) [ 1601.171571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.235919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.280884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1601.340201] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1602.400381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.451093] LNet: Removed LNI 192.168.203.5@tcp231 [ 1602.480394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.590784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1602.633514] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1602.638622] LNet: 85062:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1603.293637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1603.355412] LNet: Removed LNI 192.168.203.5@tcp231 [ 1603.390746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1603.444029] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1604.557709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.582633] LNet: 85206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.596822] LNet: 85206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1604.616030] LNet: Removed LNI 192.168.203.5@tcp231 [ 1604.639584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.719770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1604.749954] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 1604.757872] LNet: 85215:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1605.049126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1605.077337] LNet: Removed LNI 192.168.203.5@tcp231 [ 1605.104163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1605.127616] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 1606.178651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.208153] LNet: Removed LNI 192.168.203.5@tcp231 [ 1606.230140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.300144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1606.336115] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 1606.342063] LNet: 85366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1606.731473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1606.775175] LNet: Removed LNI 192.168.203.5@tcp231 [ 1606.806255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1606.849761] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 1607.723494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.759661] LNet: Removed LNI 192.168.203.5@tcp231 [ 1607.807673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.862025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1607.901606] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1608.333773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1608.369377] LNet: Removed LNI 192.168.203.5@tcp231 [ 1608.384492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1608.411228] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1614.939292] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:31:00 (1784046660) [ 1615.038675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.070670] LNetError: 86222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.082404] LNetError: 86222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1615.106770] LNet: Removed LNI 192.168.203.5@tcp231 [ 1615.142429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.229819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1615.257170] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1615.938893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1615.983717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1616.015188] LNet: Removed LNI 192.168.203.5@tcp232 [ 1616.031374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1616.057840] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1616.648981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1616.701955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1616.743150] LNet: Removed LNI 192.168.203.5@tcp232 [ 1616.761739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1616.809133] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1617.587657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1617.635139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1617.663719] LNet: Removed LNI 192.168.203.5@tcp232 [ 1617.685845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1617.722451] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1618.402535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1618.446734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1618.471216] LNet: Removed LNI 192.168.203.5@tcp232 [ 1618.490934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1624.557650] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:31:10 (1784046670) [ 1625.466725] Key type .llcrypt unregistered [ 1625.471661] Key type ._llcrypt unregistered [ 1630.199639] Key type ._llcrypt registered [ 1630.201793] Key type .llcrypt registered [ 1630.403039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1630.437042] alg: No test for adler32 (adler32-zlib) [ 1631.328437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1631.361867] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1637.638810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1646.214245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.266343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1646.305996] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1648.154693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 1648.195648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 1648.258695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 1648.357032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 1648.424425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 1648.471266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 1648.530899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 1648.583910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 1648.636213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1648.659359] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1648.671360] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1649.004201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1649.037107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1649.058022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1649.086084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1649.121379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1649.162501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1649.206781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1649.241450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1649.258784] LNet: 88018:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.277732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1649.302783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1649.322993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1649.345655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1649.374369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1649.413900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1650.283272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 1650.332594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 1650.377976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 1650.454489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 1650.527566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 1650.597813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 1650.659307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 1650.726993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 1650.798860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1650.816269] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1651.160335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1651.243322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1651.296932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1651.356049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1651.424941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1651.454623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1651.539972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1651.640626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1651.683204] LNet: 88139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1651.729883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1651.784267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1651.827538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1651.865902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1651.921491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1651.977955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1652.663729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 1652.728344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 1652.775196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 1652.856606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 1652.943969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 1653.022694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 1653.075707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 1653.156369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 1653.232594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1653.259837] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1653.554666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.588886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1653.623166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1653.660919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1653.727774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1653.763290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1653.790590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1653.821913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1653.851318] LNet: 88259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.892780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1653.943599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1654.008681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1654.068453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1654.109232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1654.148200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1654.793982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 1654.863568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 1654.922949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 1654.974251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 1655.076192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 1655.123255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 1655.183750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 1655.214290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 1655.251765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1655.281069] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1655.589804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1655.655639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1655.718878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1655.783568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1655.824994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1655.852317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1655.885333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.920889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1655.946303] LNet: 88379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.985896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.035789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.083940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.141119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.192629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.240367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1656.914376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 1656.946297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 1656.981649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 1657.024256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 1657.055566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 1657.093460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 1657.134642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 1657.168436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 1657.222985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1657.504850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.552371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1657.596381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1657.648269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1657.707896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1657.781968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1657.844553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1657.893341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1657.931350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1657.972573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1658.010219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.063417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1658.131949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1658.177858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.899830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 1658.964834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 1659.044639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 1659.145701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 1659.223564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 1659.272467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 1659.360580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 1659.417136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 1659.481193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1659.507152] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1659.513164] LNet: Skipped 1 previous similar message [ 1659.519152] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1659.756403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1659.798941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1659.842958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1659.891164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1659.920142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1659.963340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1660.018263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1660.051884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1660.083844] LNet: 88619:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.106805] LNet: 88619:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1660.148816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1660.202571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1660.253876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1660.294673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1660.344891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.408972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1665.551300] LNet: 88694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.560358] LNetError: 88694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.568840] LNet: Removed LNI 192.168.203.5@tcp [ 1665.590183] LNet: Removed LNI 192.168.203.5@tcp1 [ 1666.056631] Key type .llcrypt unregistered [ 1666.059021] Key type ._llcrypt unregistered [ 1680.346923] Key type ._llcrypt registered [ 1680.351260] Key type .llcrypt registered [ 1681.839672] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:32:07 (1784046727) [ 1682.177592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.198731] alg: No test for adler32 (adler32-zlib) [ 1683.110778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.178326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.220580] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1683.976378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1684.022105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1684.040642] LNetError: 89674:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1684.188036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.5@tcp [ 1684.192539] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1684.312953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.872617] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:32:16 (1784046736) [ 1690.972151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.017943] LNet: 90259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.026442] LNetError: 90259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.071622] LNet: Removed LNI 192.168.203.5@tcp [ 1691.111160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.206704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.225911] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1691.879748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.909196] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1692.955430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.5@tcp [ 1692.982314] LNetError: 90531:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1693.218229] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1694.264725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.5@tcp [ 1701.077526] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:32:26 (1784046746) [ 1702.709355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg305-server [ 1704.402372] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:32:30 (1784046750) [ 1706.316371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg305-server [ 1708.019970] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:32:33 (1784046753) [ 1709.959451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg305-server [ 1711.499461] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:32:37 (1784046757) [ 1711.578611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.602109] LNet: 92144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.609596] LNetError: 92144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.615674] LNet: Removed LNI 192.168.203.5@tcp [ 1711.629204] LNet: Removed LNI 192.168.203.5@tcp1 [ 1711.670221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.763396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.785729] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1712.341848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1712.391067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1712.437431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1712.535360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1712.581833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1712.620814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1718.742736] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:32:44 (1784046764) [ 1718.833951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.875625] LNet: 92872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.896893] LNetError: 92872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.905342] LNetError: 92872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.943513] LNet: Removed LNI 192.168.203.5@tcp [ 1718.971533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.030292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.056370] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1726.547862] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:32:51 (1784046771) [ 1726.902568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.943965] LNet: 93580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.960712] LNetError: 93580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.997308] LNet: Removed LNI 192.168.203.5@tcp [ 1727.030487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.132162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.161390] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1735.436412] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:33:01 (1784046781) [ 1737.549644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1739.441408] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:33:05 (1784046785) [ 1740.244605] LNet: 94617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.253140] LNetError: 94617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.269631] LNet: Removed LNI 192.168.203.5@tcp [ 1740.948854] Key type .llcrypt unregistered [ 1740.951834] Key type ._llcrypt unregistered [ 1746.900303] Key type ._llcrypt registered [ 1746.903241] Key type .llcrypt registered [ 1747.064807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.076298] alg: No test for adler32 (adler32-zlib) [ 1748.021931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1748.063269] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1755.488787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1761.506967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.105@tcp [ 1761.954395] LNet: 95034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.969580] LNetError: 95034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.011996] LNet: Removed LNI 192.168.203.5@tcp [ 1762.640132] Key type .llcrypt unregistered [ 1762.645589] Key type ._llcrypt unregistered [ 1773.750982] Key type ._llcrypt registered [ 1773.760440] Key type .llcrypt registered [ 1775.006202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1776.528594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1778.262198] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:33:43 (1784046823) [ 1779.576542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1781.475297] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:33:47 (1784046827) [ 1783.496186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg305-server [ 1785.032627] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:33:50 (1784046830) [ 1787.016346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg305-server [ 1788.889775] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:33:54 (1784046834) [ 1790.410477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.475039] alg: No test for adler32 (adler32-zlib) [ 1791.448892] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1791.625865] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1793.280117] Key type lgssc registered [ 1794.322587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.937866] Key type lgssc unregistered [ 1813.159970] LNet: 98209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.167868] LNetError: 98209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.181458] LNet: Removed LNI 192.168.203.5@tcp [ 1813.806587] Key type .llcrypt unregistered [ 1813.817236] Key type ._llcrypt unregistered [ 1828.042167] Key type ._llcrypt registered [ 1828.044609] Key type .llcrypt registered [ 1829.327633] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:34:35 (1784046875) [ 1829.702653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.721567] alg: No test for adler32 (adler32-zlib) [ 1830.718694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.804598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1830.905624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1830.962749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1831.018294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1837.376987] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:34:43 (1784046883) [ 1837.554192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.586290] LNet: 99624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.618242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.665890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.694089] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1838.607974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.626858] LNet: 99769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.632832] LNetError: 99769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.657382] LNet: Removed LNI 192.168.203.5@tcp [ 1838.676476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.740916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1838.773237] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1839.170865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.186964] LNetError: 99797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.204698] LNet: Removed LNI 192.168.203.5@tcp [ 1839.226483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.299536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1839.325763] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1846.413238] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:34:52 (1784046892) [ 1846.803610] LNet: 100388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.813633] LNet: 100388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1846.827769] LNetError: 100388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.861773] LNet: Removed LNI 192.168.203.5@tcp [ 1847.519844] Key type .llcrypt unregistered [ 1847.522119] Key type ._llcrypt unregistered [ 1853.882700] Key type ._llcrypt registered [ 1853.889838] Key type .llcrypt registered [ 1854.164297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1854.182250] alg: No test for adler32 (adler32-zlib) [ 1855.108260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.189538] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1855.195065] LNet: *** cfs_fail_loc=e003, val=100*** [ 1856.297172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1856.321522] LNet: *** cfs_fail_loc=e003, val=100*** [ 1856.328531] LNet: Skipped 2 previous similar messages [ 1856.333976] Ping reply received with mapping: [ 1856.335839] found metadata reply NI[192.168.203.5@tcp] [ 1856.338290] 05 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1856.341579] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1856.343791] 5a 5a 5a 5a 5a 5a 5a 5a [ 1856.623480] LNet: 100785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.629266] LNetError: 100785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.640856] LNet: Removed LNI 192.168.203.5@tcp [ 1858.461049] Key type .llcrypt unregistered [ 1858.466644] Key type ._llcrypt unregistered [ 1871.969100] Key type ._llcrypt registered [ 1871.970458] Key type .llcrypt registered [ 1873.431882] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:35:18 (1784046918) [ 1873.969582] Key type .llcrypt unregistered [ 1873.975541] Key type ._llcrypt unregistered [ 1880.445412] Key type ._llcrypt registered [ 1880.450262] Key type .llcrypt registered [ 1880.685968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.701236] alg: No test for adler32 (adler32-zlib) [ 1881.893734] LNetError: 101832:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1882.311546] Key type .llcrypt unregistered [ 1882.315292] Key type ._llcrypt unregistered [ 1888.236635] Key type ._llcrypt registered [ 1888.243181] Key type .llcrypt registered [ 1889.945683] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:35:35 (1784046935) [ 1890.339731] Key type .llcrypt unregistered [ 1890.343580] Key type ._llcrypt unregistered [ 1896.744739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1896.748662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1899.269794] Key type ._llcrypt registered [ 1899.275807] Key type .llcrypt registered [ 1899.541705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1899.559830] alg: No test for adler32 (adler32-zlib) [ 1900.498192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.581163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1909.672982] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:35:55 (1784046955) [ 1909.826951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.863335] LNet: 103610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.884531] LNetError: 103610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1909.918406] LNet: Removed LNI 10.1.2.3@tcp [ 1909.946626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.790593] Key type .llcrypt unregistered [ 1910.794341] Key type ._llcrypt unregistered [ 1917.263566] Key type ._llcrypt registered [ 1917.266226] Key type .llcrypt registered [ 1917.509205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1917.524326] alg: No test for adler32 (adler32-zlib) [ 1918.452601] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1918.533389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1918.690844] LNet: 103901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.704333] LNetError: 103901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.791879] LNet: Removed LNI 192.168.203.5@tcp [ 1920.518476] Key type .llcrypt unregistered [ 1920.524543] Key type ._llcrypt unregistered [ 1931.481684] Key type ._llcrypt registered [ 1931.487319] Key type .llcrypt registered [ 1932.971149] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:36:18 (1784046978) [ 1933.733506] Key type .llcrypt unregistered [ 1933.735311] Key type ._llcrypt unregistered [ 1940.429558] Key type ._llcrypt registered [ 1940.435498] Key type .llcrypt registered [ 1940.733960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1940.775783] alg: No test for adler32 (adler32-zlib) [ 1948.811919] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:36:34 (1784046994) [ 1948.951513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.022618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.070369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.121044] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1950.014549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.065878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1950.092927] LNet: Removed LNI 192.168.203.5@tcp [ 1950.308250] LNet: 105646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.639641] Key type .llcrypt unregistered [ 1950.644215] Key type ._llcrypt unregistered [ 1962.567737] Key type ._llcrypt registered [ 1962.569957] Key type .llcrypt registered [ 1963.976327] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:36:49 (1784047009) [ 1964.555510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.584961] alg: No test for adler32 (adler32-zlib) [ 1965.536411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.704824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.763769] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1966.671897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1966.731354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.5@tcp [ 1966.761992] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1966.768734] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1966.772409] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1966, 0, 0 [ 1966.782738] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1966.786191] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1966.795626] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1966.799714] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1966, 0, 0 [ 1966.808300] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1966.811456] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1966.815401] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1966.824412] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1966, 0, 2016 [ 1966.831302] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1966.838124] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1966.843985] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1966.850144] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1966, 0, 2016 [ 1966.856470] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1973.124726] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:36:58 (1784047018) [ 1973.524658] LNet: 107186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.530244] LNetError: 107186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.548406] LNet: Removed LNI 192.168.203.5@tcp [ 1974.192481] Key type .llcrypt unregistered [ 1974.196840] Key type ._llcrypt unregistered [ 1982.138528] Key type ._llcrypt registered [ 1982.144031] Key type .llcrypt registered [ 1982.390612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1982.430428] alg: No test for adler32 (adler32-zlib) [ 1983.315482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1983.336696] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1990.880191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 2000.383424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.461176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2000.521118] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 2001.901267] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2002.207197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2002.213250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2002.228530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2004.617468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2004.641225] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2014.955398] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:37:41 (1784047061) [ 2015.321575] LNet: 108698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.331593] LNetError: 108698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.335488] LNet: Removed LNI 192.168.203.5@tcp [ 2016.352320] LNet: Removed LNI 192.168.203.5@tcp1 [ 2016.355954] LNetError: 108698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2016.363731] LNetError: 108698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2016.388352] LNet: Removed LNI 10.1.2.3@tcp99 [ 2016.933754] Key type .llcrypt unregistered [ 2016.935947] Key type ._llcrypt unregistered [ 2023.862749] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2023.868533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2026.600507] Key type ._llcrypt registered [ 2026.605772] Key type .llcrypt registered [ 2026.839544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.852893] alg: No test for adler32 (adler32-zlib) [ 2027.826489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.909378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.956738] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2028.875829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2028.907816] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2030.943116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2031.002481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2031.060334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2031.093993] LNetError: 109442:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2031.127339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2031.177863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2031.650613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2031.760262] LNet: 109462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.773691] LNetError: 109462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.778477] LNet: Removed LNI 10.1.2.3@tcp [ 2032.809195] LNet: Removed LNI 192.168.203.5@tcp [ 2033.502983] Key type .llcrypt unregistered [ 2033.507828] Key type ._llcrypt unregistered [ 2046.352412] Key type ._llcrypt registered [ 2046.355199] Key type .llcrypt registered [ 2047.936336] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:38:13 (1784047093) [ 2048.678607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.698528] alg: No test for adler32 (adler32-zlib) [ 2049.663056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.712365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.755753] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2050.387630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 2050.475707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg305-client@tcp [ 2056.782944] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:38:22 (1784047102) [ 2056.921382] LNet: 110990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.928718] LNetError: 110990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.956154] LNet: Removed LNI 192.168.203.5@tcp [ 2057.761601] Key type .llcrypt unregistered [ 2057.764816] Key type ._llcrypt unregistered [ 2064.124401] Key type ._llcrypt registered [ 2064.126936] Key type .llcrypt registered [ 2064.370549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.385833] alg: No test for adler32 (adler32-zlib) [ 2065.410685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2065.439489] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2065.497634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2073.467834] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:38:38 (1784047118) [ 2074.917670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2078.107223] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:38:42 (1784047122) [ 2079.818411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2082.129126] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:38:47 (1784047127) [ 2082.402517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2082.445455] LNet: 112417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.466808] LNetError: 112417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.491427] LNet: Removed LNI 192.168.203.5@tcp [ 2082.521167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.637762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2082.684040] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2083.864236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2083.963613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2092.861453] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:38:58 (1784047138) [ 2092.966266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.999358] LNet: 113124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.017867] LNetError: 113124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.051134] LNet: Removed LNI 192.168.203.5@tcp [ 2093.081152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.126996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2093.154528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2093.815395] Key type .llcrypt unregistered [ 2093.818501] Key type ._llcrypt unregistered [ 2105.399925] Key type ._llcrypt registered [ 2105.401591] Key type .llcrypt registered [ 2106.895887] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:39:12 (1784047152) [ 2107.415295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.438516] alg: No test for adler32 (adler32-zlib) [ 2108.391809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.470629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.502897] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2109.047911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2109.089665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2115.689857] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:39:21 (1784047161) [ 2115.918847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.948161] LNet: 114686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.962137] LNetError: 114686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.978587] LNet: Removed LNI 192.168.203.5@tcp [ 2116.015876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.107829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2116.176249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2122.570291] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:39:28 (1784047168) [ 2122.757646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.785675] LNet: 115266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.829479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.942482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2123.487587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2123.524680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2123.602378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2124.136161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2124.204765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2124.304896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.326341] LNet: 115300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.351684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.443279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2124.908656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2124.954332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2125.029334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2125.487161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2125.525929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2131.446576] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:39:37 (1784047177) [ 2131.568924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.630181] LNet: 115894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.689665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.749453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.774543] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2132.690671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2132.718800] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2133.481526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2133.555742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2133.808314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.839489] LNetError: 116179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.852728] LNet: Removed LNI 192.168.203.5@tcp [ 2133.874879] LNet: Removed LNI 192.168.203.5@tcp2 [ 2133.909733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.967750] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2133.972833] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2133.978074] LNet: 116185:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2134.649951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2134.890410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.919618] LNetError: 116222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.923954] LNetError: 116222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2135.968149] LNet: Removed LNI 192.168.203.5@tcp [ 2135.990366] LNet: Removed LNI 192.168.203.5@tcp2 [ 2136.020226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.093812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.135330] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2136.766647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2136.810253] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2137.816927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2137.847977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2137.895656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2138.107457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.139147] LNet: 116507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.157525] LNet: 116507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2138.168723] LNetError: 116507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.177814] LNetError: 116507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2138.182139] LNet: Removed LNI 192.168.203.5@tcp [ 2139.239238] LNet: Removed LNI 192.168.203.5@tcp2 [ 2139.282826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.352798] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2139.358657] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2139.364586] LNet: 116513:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2139.854571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2140.046512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.069206] LNet: Removed LNI 192.168.203.5@tcp [ 2141.097343] LNet: Removed LNI 192.168.203.5@tcp2 [ 2141.149459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.237561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.269946] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2141.827258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.847763] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2142.762375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2142.805265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2143.023707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.056821] LNetError: 116830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.063870] LNetError: 116830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2143.077625] LNet: Removed LNI 192.168.203.5@tcp [ 2143.087520] LNet: Removed LNI 192.168.203.5@tcp2 [ 2143.110326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.187526] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2143.193892] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2143.200968] LNet: 116836:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2143.768360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2143.946983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.992326] LNet: Removed LNI 192.168.203.5@tcp [ 2145.001528] LNet: Removed LNI 192.168.203.5@tcp2 [ 2145.060854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.130194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.163380] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2146.007150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2146.023337] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2146.809661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2146.839697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2146.888960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2147.098717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.120672] LNet: 117157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.132130] LNet: 117157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2148.193273] LNet: Removed LNI 192.168.203.5@tcp [ 2149.232626] LNet: Removed LNI 192.168.203.5@tcp2 [ 2149.257065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.331793] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2149.339817] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2149.348182] LNet: 117164:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2149.870392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2150.078232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.095411] LNet: Removed LNI 192.168.203.5@tcp [ 2151.151483] LNet: Removed LNI 192.168.203.5@tcp2 [ 2151.175600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.270927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.294781] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2151.864405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2151.895778] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2152.539207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2152.568810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2152.618615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2152.906427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.931018] LNetError: 117498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.941231] LNetError: 117498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2152.948610] LNet: Removed LNI 192.168.203.5@tcp [ 2152.965299] LNet: Removed LNI 192.168.203.5@tcp2 [ 2152.996275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.050587] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2153.056231] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2153.582362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2153.917804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.976249] LNet: Removed LNI 192.168.203.5@tcp [ 2154.992858] LNet: Removed LNI 192.168.203.5@tcp2 [ 2155.014156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.078872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.116589] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2155.768204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2155.784686] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2156.546545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2156.579515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2156.613868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2156.670357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2156.952449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.984433] LNet: Removed LNI 192.168.203.5@tcp [ 2159.014902] LNet: Removed LNI 192.168.203.5@tcp2 [ 2159.056603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.105321] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2159.112623] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2159.117921] LNet: 117864:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2159.123609] LNet: 117864:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2159.537647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2166.711864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2168.675141] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:40:14 (1784047214) [ 2168.839503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.898454] LNet: 118618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.908990] LNet: 118618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2168.921504] LNet: Removed LNI 192.168.203.5@tcp [ 2168.945041] LNetError: 118618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.954403] LNetError: 118618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2168.982994] LNet: Removed LNI 192.168.203.5@tcp2 [ 2169.026258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.312069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2169.348846] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2170.212392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2170.236322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2171.359043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.393316] LNet: Removed LNI 10.1.2.3@tcp [ 2171.408045] LNet: Removed LNI 192.168.203.5@tcp [ 2171.445476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.511077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2171.535886] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2171.546602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2172.163180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2172.188536] LNet: Removed LNI 192.168.203.5@tcp [ 2172.206307] LNet: Removed LNI 10.1.2.3@tcp [ 2177.948420] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:40:23 (1784047223) [ 2178.169406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.202534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.318233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2178.419822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2178.422915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2178.426623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2180.790605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.813157] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2181.467631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2181.487667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2183.545841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2183.592959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2183.675593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.5@tcp -m PUT -e local_timeout -r 1 [ 2183.758917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2183.846828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2183.851181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.135413] LNetError: 119565:0:(lib-move.c:4771:lnet_parse()) 192.168.203.5@tcp, src 192.168.203.5@tcp: Dropping PUT (error -108 looking up sender) [ 2186.208233] LNet: Removed LNI 10.1.2.3@tcp [ 2187.239804] LNet: Removed LNI 192.168.203.5@tcp [ 2187.650141] Key type .llcrypt unregistered [ 2187.652299] Key type ._llcrypt unregistered [ 2199.591157] Key type ._llcrypt registered [ 2199.593323] Key type .llcrypt registered [ 2201.029804] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:40:46 (1784047246) [ 2201.461900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.498882] alg: No test for adler32 (adler32-zlib) [ 2202.415084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.830553] LNet: 120877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.499564] Key type .llcrypt unregistered [ 2203.503659] Key type ._llcrypt unregistered [ 2210.069490] Key type ._llcrypt registered [ 2210.074699] Key type .llcrypt registered [ 2210.313635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2210.341357] alg: No test for adler32 (adler32-zlib) [ 2211.260836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2211.295416] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2219.394700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 2219.609329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2220.363285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 2220.512317] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 2220.627438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2220.660307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2220.708839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2220.762712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2220.797204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2220.854064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.890772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.916613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.935038] LNet: 121327:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.958211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.003519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.059786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.102762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.139800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.172564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.900353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2222.160668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2222.189723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2222.215379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2222.243391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2222.274565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2222.306325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.336410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.368603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.390485] LNet: 121444:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.412341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.439496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.469446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.511304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.573407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.623577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.482288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 2223.782692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.840711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2223.888930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.944535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2224.030711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2224.109071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2224.153298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2224.191175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2224.212145] LNet: 121562:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2224.245179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2224.275320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2224.306976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2224.338366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2224.370788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2224.403873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2224.921365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2225.178781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2225.221461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2225.256766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2225.284740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2225.326176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2225.369375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2225.426626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2225.453266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2225.492642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2225.543945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2225.589555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2225.634424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2225.675930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2225.722347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2225.994482] LNet: 121725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.006227] LNetError: 121725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.025550] LNet: Removed LNI 192.168.203.5@tcp [ 2226.368622] Key type .llcrypt unregistered [ 2226.370790] Key type ._llcrypt unregistered [ 2242.027217] Key type ._llcrypt registered [ 2242.029917] Key type .llcrypt registered [ 2243.257231] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:41:29 (1784047289) [ 2243.742562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2243.755628] alg: No test for adler32 (adler32-zlib) [ 2244.662174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.942890] LNet: 122613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.460361] Key type .llcrypt unregistered [ 2245.465080] Key type ._llcrypt unregistered [ 2251.620645] Key type ._llcrypt registered [ 2251.626505] Key type .llcrypt registered [ 2251.800832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2251.829146] alg: No test for adler32 (adler32-zlib) [ 2252.763097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2252.801258] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2260.118885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 2268.322119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.367244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2268.397890] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 2269.131311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2269.742742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 2270.005636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.034162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.058955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.090812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.123393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.158986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.199880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.238335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.241291] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 2270.264178] LNet: 123266:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.300471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.354078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2270.392214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2270.428992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2270.489183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2270.528744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2271.154690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2271.400260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2271.431457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2271.459687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2271.482925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2271.515874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2271.552901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2271.586164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2271.638425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2271.656458] LNet: 123383:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.678156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2271.707565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.754912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.803716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.855395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.905705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2272.441071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 2272.615474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.645628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2272.680070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.714508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2272.749961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2272.778693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2272.808686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2272.858422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2272.877145] LNet: 123500:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.898443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2272.930143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2272.970504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2273.011735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2273.040777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2273.073435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2273.690762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2273.900318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2273.937297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2273.968694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2274.000147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.033703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.081915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2274.126821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2274.187336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2274.231747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2274.279545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2274.316426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2274.345505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2274.397121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2274.428835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2279.684770] LNet: 123698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.690372] LNetError: 123698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.697176] LNet: Removed LNI 192.168.203.5@tcp [ 2279.717925] LNet: Removed LNI 192.168.203.5@tcp1 [ 2280.120645] Key type .llcrypt unregistered [ 2280.123353] Key type ._llcrypt unregistered [ 2295.474220] Key type ._llcrypt registered [ 2295.479321] Key type .llcrypt registered [ 2296.648886] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:42:22 (1784047342) [ 2298.185688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2300.073791] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:42:25 (1784047345) [ 2308.776803] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:42:34 (1784047354) [ 2317.896137] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:42:43 (1784047363) [ 2326.062572] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:42:51 (1784047371) [ 2335.431457] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:43:00 (1784047380) [ 2343.554132] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:43:09 (1784047389) [ 2351.362371] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:43:17 (1784047397) [ 2359.638876] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:43:25 (1784047405) [ 2368.574329] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:43:34 (1784047414) [ 2375.056695] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:43:40 (1784047420) [ 2381.613266] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:43:47 (1784047427) [ 2389.441341] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:43:55 (1784047435) [ 2396.842485] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:44:02 (1784047442) [ 2403.967781] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:44:09 (1784047449) [ 2412.613730] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:44:17 (1784047457) [ 2421.516875] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:44:27 (1784047467) [ 2430.553968] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:44:35 (1784047475) [ 2440.366645] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:44:46 (1784047486) [ 2440.911911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.933385] alg: No test for adler32 (adler32-zlib) [ 2441.891198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.084580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.131166] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2443.276706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.300533] LNet: 134701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.315953] LNetError: 134701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.329921] LNet: Removed LNI 192.168.203.5@tcp [ 2443.343982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.388209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2443.411560] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2450.746829] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:44:56 (1784047496) [ 2459.357722] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:45:04 (1784047504) [ 2468.443128] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:45:14 (1784047514) [ 2476.639448] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:45:22 (1784047522) [ 2484.199838] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:45:29 (1784047529) [ 2492.609665] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:45:38 (1784047538) [ 2499.765486] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:45:45 (1784047545) [ 2506.942161] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:45:52 (1784047552) [ 2515.274119] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:46:00 (1784047560) [ 2522.848162] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:46:08 (1784047568) [ 2529.693712] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:46:15 (1784047575) [ 2537.005129] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:46:22 (1784047582) [ 2543.750780] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2321 sec ========== 12:46:29 (1784047589) [ 2544.204499] LNet: 142168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.213353] LNetError: 142168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.231783] LNet: Removed LNI 192.168.203.5@tcp [ 2544.844717] Key type .llcrypt unregistered [ 2544.846415] Key type ._llcrypt unregistered [ 2557.010387] Key type ._llcrypt registered [ 2557.015856] Key type .llcrypt registered [ 2557.583280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2557.593967] alg: No test for adler32 (adler32-zlib) [ 2558.608134] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2558.821639] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2560.464506] Key type lgssc registered [ 2561.783244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2624.664603] Lustre: Mounted lustre-client [ 2630.558290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2640.122744] Lustre: Unmounted lustre-client [ 2661.286491] Key type lgssc unregistered [ 2661.658953] LNet: 145095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.677974] LNetError: 145095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2661.698786] LNet: Removed LNI 192.168.203.5@tcp [ 2662.620233] Key type .llcrypt unregistered [ 2662.622387] Key type ._llcrypt unregistered