[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 404988277 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002352] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006907] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.010054] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012044] *** VALIDATE selinux *** [ 0.019015] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023731] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025107] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] *** VALIDATE tmpfs *** [ 0.030258] *** VALIDATE proc *** [ 0.031241] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039815] debug: unmapping init [mem 0xffffffffbb259000-0xffffffffbb260fff] [ 0.041978] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.045840] ... value mask: 0000ffffffffffff [ 0.046016] ... max period: 00007fffffffffff [ 0.046959] ... fixed-purpose events: 3 [ 0.047013] ... event mask: 000000070000000f [ 0.048308] rcu: Hierarchical SRCU implementation. [ 0.050373] smp: Bringing up secondary CPUs ... [ 0.051488] x86: Booting SMP configuration: [ 0.052025] .... node #0, CPUs: #1 #2 #3 [ 0.055195] smp: Brought up 1 node, 4 CPUs [ 0.057016] smpboot: Max logical packages: 1 [ 0.058011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125422] node 0 deferred pages initialised in 63ms [ 0.128527] devtmpfs: initialized [ 0.129149] x86/mm: Memory block size: 128MB [ 0.131232] gcov: version magic: 0x41383552 [ 0.133275] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134050] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135241] pinctrl core: initialized pinctrl subsystem [ 0.136180] [ 0.136564] ************************************************************* [ 0.137011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138010] ** ** [ 0.139009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140008] ** ** [ 0.141012] ** This means that this kernel is built to expose internal ** [ 0.142013] ** IOMMU data structures, which may compromise security on ** [ 0.143007] ** your system. ** [ 0.144014] ** ** [ 0.145012] ** If you see this message and you are not debugging the ** [ 0.146009] ** kernel, report this immediately to your vendor! ** [ 0.147008] ** ** [ 0.148012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149012] ************************************************************* [ 0.150565] NET: Registered protocol family 16 [ 0.151464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153122] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.154489] cpuidle: using governor menu [ 0.156662] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159461] PCI: Using configuration type 1 for base access [ 0.161132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175054] cryptd: max_cpu_qlen set to 1000 [ 0.177239] ACPI: Added _OSI(Module Device) [ 0.178008] ACPI: Added _OSI(Processor Device) [ 0.178825] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.183200] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188541] ACPI: Interpreter enabled [ 0.189058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190013] ACPI: Using IOAPIC for interrupt routing [ 0.191134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.192438] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.203121] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.204037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.205019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.206086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.208417] acpiphp: Slot [2] registered [ 0.209116] acpiphp: Slot [5] registered [ 0.209990] acpiphp: Slot [6] registered [ 0.210118] acpiphp: Slot [3] registered [ 0.210979] acpiphp: Slot [4] registered [ 0.211092] acpiphp: Slot [7] registered [ 0.211976] acpiphp: Slot [8] registered [ 0.212097] acpiphp: Slot [9] registered [ 0.213135] acpiphp: Slot [10] registered [ 0.214108] acpiphp: Slot [11] registered [ 0.215098] acpiphp: Slot [12] registered [ 0.215995] acpiphp: Slot [13] registered [ 0.216085] acpiphp: Slot [14] registered [ 0.216966] acpiphp: Slot [15] registered [ 0.217052] acpiphp: Slot [16] registered [ 0.217876] acpiphp: Slot [17] registered [ 0.219071] acpiphp: Slot [18] registered [ 0.219680] acpiphp: Slot [19] registered [ 0.220048] acpiphp: Slot [20] registered [ 0.220820] acpiphp: Slot [21] registered [ 0.222063] acpiphp: Slot [22] registered [ 0.222958] acpiphp: Slot [23] registered [ 0.223102] acpiphp: Slot [24] registered [ 0.224023] acpiphp: Slot [25] registered [ 0.224907] acpiphp: Slot [26] registered [ 0.226062] acpiphp: Slot [27] registered [ 0.226985] acpiphp: Slot [28] registered [ 0.228068] acpiphp: Slot [29] registered [ 0.229085] acpiphp: Slot [30] registered [ 0.229929] acpiphp: Slot [31] registered [ 0.231061] PCI host bridge to bus 0000:00 [ 0.232065] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.233020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.235043] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.236017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.238017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.239034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.241189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.242858] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.244886] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.249830] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.253000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.253000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.255012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.257014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.258396] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.260893] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.262033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.264456] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.267804] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.275012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.278011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.282940] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.294036] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.305014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.320017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.326615] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.331012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.336011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.346014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.354927] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.357345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.358241] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.360297] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.362269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.366023] iommu: Default domain type: Passthrough [ 0.367402] SCSI subsystem initialized [ 0.369137] ACPI: bus type USB registered [ 0.370100] usbcore: registered new interface driver usbfs [ 0.371038] usbcore: registered new interface driver hub [ 0.372061] usbcore: registered new device driver usb [ 0.373121] pps_core: LinuxPPS API ver. 1 registered [ 0.375006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.376070] PTP clock support registered [ 0.378164] EDAC MC: Ver: 3.0.0 [ 0.379087] PCI: Using ACPI for IRQ routing [ 0.380627] NetLabel: Initializing [ 0.381010] NetLabel: domain hash size = 128 [ 0.383008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.384059] NetLabel: unlabeled traffic allowed by default [ 0.385296] vgaarb: loaded [ 0.386285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.388009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.393447] clocksource: Switched to clocksource kvm-clock [ 0.503688] VFS: Disk quotas dquot_6.6.0 [ 0.505071] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.507437] *** VALIDATE ramfs *** [ 0.508595] *** VALIDATE hugetlbfs *** [ 0.510807] pnp: PnP ACPI init [ 0.513183] pnp: PnP ACPI: found 6 devices [ 0.528308] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.531277] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.533413] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.535548] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.537681] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.539664] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.541990] NET: Registered protocol family 2 [ 0.544288] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.548626] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.552197] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.557384] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.562666] TCP: Hash tables configured (established 65536 bind 65536) [ 0.565617] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.568618] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.571215] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.573914] NET: Registered protocol family 1 [ 0.576262] RPC: Registered named UNIX socket transport module. [ 0.578175] RPC: Registered udp transport module. [ 0.579607] RPC: Registered tcp transport module. [ 0.581093] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.582952] NET: Registered protocol family 44 [ 0.584323] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.585988] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.587526] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.589710] PCI: CLS 0 bytes, default 64 [ 0.591376] Unpacking initramfs... [ 1.942282] debug: unmapping init [mem 0xffff9bc67cc64000-0xffff9bc67ffcffff] [ 1.946078] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.948302] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.951043] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.445256] Initialise system trusted keyrings [ 2.446766] Key type blacklist registered [ 2.449088] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.457384] zbud: loaded [ 2.460392] *** VALIDATE nfs *** [ 2.461468] *** VALIDATE nfs4 *** [ 2.463046] pstore: using deflate compression [ 2.466466] Platform Keyring initialized [ 2.558642] NET: Registered protocol family 38 [ 2.559956] Key type asymmetric registered [ 2.560956] Asymmetric key parser 'x509' registered [ 2.561932] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.563608] io scheduler mq-deadline registered [ 2.564688] io scheduler kyber registered [ 2.565660] io scheduler bfq registered [ 2.566724] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.568436] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.570454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.572890] ACPI: Power Button [PWRF] [ 2.577633] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.583432] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.591444] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.621077] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.657056] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.661260] Non-volatile memory driver v1.3 [ 2.662340] Linux agpgart interface v0.103 [ 2.690901] virtio_blk virtio1: [vda] 134584 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.694027] vda: detected capacity change from 0 to 68907008 [ 2.706328] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.708532] vdb: detected capacity change from 0 to 1073741824 [ 2.713145] libphy: Fixed MDIO Bus: probed [ 2.727082] usbcore: registered new interface driver usbserial_generic [ 2.729254] usbserial: USB Serial support registered for generic [ 2.731274] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.734990] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.736516] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.738980] mousedev: PS/2 mouse device common for all mice [ 2.743247] rtc_cmos 00:05: RTC can wake from S4 [ 2.745932] rtc_cmos 00:05: registered as rtc0 [ 2.747632] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.750346] intel_pstate: CPU model not supported [ 2.751086] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.755714] hid: raw HID events driver (C) Jiri Kosina [ 2.757127] usbcore: registered new interface driver usbhid [ 2.759740] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.760050] usbhid: USB HID core driver [ 2.764151] drop_monitor: Initializing network drop monitor service [ 2.764325] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.765886] Initializing XFRM netlink socket [ 2.770686] NET: Registered protocol family 10 [ 2.773594] Segment Routing with IPv6 [ 2.774886] NET: Registered protocol family 17 [ 2.777310] mpls_gso: MPLS GSO support [ 2.784408] RAS: Correctable Errors collector initialized. [ 2.785848] AVX version of gcm_enc/dec engaged. [ 2.786856] AES CTR mode by8 optimization enabled [ 2.860775] sched_clock: Marking stable (2860751848, 0)->(3751428365, -890676517) [ 2.863098] registered taskstats version 1 [ 2.864690] Loading compiled-in X.509 certificates [ 2.866473] zswap: loaded using pool lzo/zbud [ 2.890093] Key type big_key registered [ 2.902775] Key type encrypted registered [ 2.904153] ima: No TPM chip found, activating TPM-bypass! [ 2.905839] ima: Allocated hash algorithm: sha1 [ 2.907527] ima: No architecture policies found [ 2.909343] evm: Initialising EVM extended attributes: [ 2.911075] evm: security.selinux [ 2.912182] evm: security.ima [ 2.913180] evm: security.capability [ 2.914233] evm: HMAC attrs: 0x1 [ 2.917142] rtc_cmos 00:05: setting system clock to 2026-03-15 21:20:12 UTC (1773609612) [ 2.923915] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 2.926879] debug: unmapping init [mem 0xffffffffbaf82000-0xffffffffbb258fff] [ 2.935078] Write protecting the kernel read-only data: 28672k [ 2.937109] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 2.938655] debug: unmapping init [mem 0xffffffffb9f14000-0xffffffffb9ffffff] [ 2.966573] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.971707] systemd[1]: Detected virtualization kvm. [ 2.972876] systemd[1]: Detected architecture x86-64. [ 2.974154] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.997400] systemd[1]: No hostname configured. [ 2.998719] systemd[1]: Set hostname to . [ 3.000575] random: systemd: uninitialized urandom read (16 bytes read) [ 3.003041] systemd[1]: Initializing machine ID from random generator. [ 3.119021] random: systemd: uninitialized urandom read (16 bytes read) [ 3.120677] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.124040] random: systemd: uninitialized urandom read (16 bytes read) [ 3.126983] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.135573] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.747533] device-mapper: uevent: version 1.0.3 [ 3.749456] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Ke[ 3.983554] random: fast init done rnel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.531655] virtio_net virtio0 ens2: renamed from eth0 [ 4.570929] scsi host0: ata_piix [ 4.641635] scsi host1: ata_piix [ 4.643327] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.645793] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.587894] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.420326] random: crng init done [ 9.421642] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.010173] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.206637] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.448264] SELinux: Disabled at runtime. [ 11.505361] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.512702] systemd[1]: Detected virtualization kvm. [ 11.514477] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.956879] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.960836] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.966665] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.971538] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.975432] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.982841] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.987827] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 12.098399] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.503944] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.769474] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.911559] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.982393] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.995341] EDAC sbridge: Ver: 1.1.2 [ 14.011394] Key type dns_resolver registered [ 14.311312] NFS: Registering the id_resolver key type [ 14.313199] Key type id_resolver registered [ 14.314653] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-client login: [ 48.585245] libcfs: loading out-of-tree module taints kernel. [ 48.705679] Key type ._llcrypt registered [ 48.707611] Key type .llcrypt registered [ 49.718364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 49.735792] alg: No test for adler32 (adler32-zlib) [ 51.301448] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 52.302233] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 54.193570] Key type lgssc registered [ 56.192881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.750168] hrtimer: interrupt took 4127089 ns [ 201.818159] Lustre: Mounted lustre-client [ 207.321911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.300240] Lustre: lustre-OST0000-osc-ffff9bc6d8380800: disconnect after 23s idle [ 228.106572] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing check_logdir /tmp/testlogs/ [ 234.033110] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing yml_node [ 238.661409] Lustre: DEBUG MARKER: Client: 2.17.51.1 [ 241.482628] Lustre: DEBUG MARKER: MDS: 2.17.51.1 [ 244.750280] Lustre: DEBUG MARKER: OSS: 2.17.51.1 [ 246.417942] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Mar 15 17:24:14 EDT 2026 [ 264.623905] LustreError: 7914:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9bc6d8380800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 264.672837] Lustre: Unmounted lustre-client [ 294.776257] Key type lgssc unregistered [ 295.197271] LNet: 8363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.200490] LNetError: 8363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.240759] LNet: Removed LNI 192.168.204.15@tcp [ 296.035232] Key type .llcrypt unregistered [ 296.040324] Key type ._llcrypt unregistered [ 323.735684] Key type ._llcrypt registered [ 323.740715] Key type .llcrypt registered [ 324.062083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.109481] alg: No test for adler32 (adler32-zlib) [ 325.130640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.263097] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 325.310654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 329.773447] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 329.995848] LNet: 9244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.009894] LNetError: 9244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.051533] LNet: Removed LNI 192.168.204.15@tcp [ 330.950563] Key type .llcrypt unregistered [ 330.953963] Key type ._llcrypt unregistered [ 337.519983] Key type ._llcrypt registered [ 337.527636] Key type .llcrypt registered [ 339.106646] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:25:46 (1773609946) [ 339.625366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.640837] alg: No test for adler32 (adler32-zlib) [ 340.635982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.716727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.746339] LNet: 9656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.778289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.850974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 340.878552] LNet: 9663:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 348.850516] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:25:56 (1773609956) [ 348.923403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.981745] LNet: 10234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.026372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.200944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 357.354723] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:26:05 (1773609965) [ 357.679163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.698190] LNet: 10826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.736573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.836655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 365.639249] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:26:13 (1773609973) [ 365.739778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.787677] LNet: 11404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.856322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.026326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 374.519492] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:26:22 (1773609982) [ 374.851337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.891933] LNet: 11997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.920949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.015886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 375.590260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 383.232460] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:26:31 (1773609991) [ 383.522916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.545114] LNet: 12588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.560739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.665363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.174106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.731195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.696208] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:26:40 (1773610000) [ 392.803860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.883460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.055264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.341215] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:26:49 (1773610009) [ 401.460446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.488610] LNet: 13763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.505937] LNet: 13763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 401.564244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.694638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 401.747576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.358267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 402.404619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.110594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.208367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.615347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 403.652751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.045443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 404.074728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.825661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 404.892748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.518354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 405.559132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.061754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 406.104817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.701189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 406.765036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 407.318235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 407.363671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.118131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 408.176594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.727666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 408.780139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.259348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 409.310321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 416.774570] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:27:04 (1773610024) [ 417.084355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.137393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.216635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 417.248400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 424.857565] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:27:12 (1773610032) [ 424.985930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.052448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.210711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 425.254156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 433.648894] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:27:21 (1773610041) [ 434.028486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.059644] LNet: 15625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.071521] LNet: 15625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 434.110288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.194310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 434.266754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 442.767781] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:27:30 (1773610050) [ 442.996433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.071403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.258831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 443.313745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 451.616587] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:27:39 (1773610059) [ 451.770617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.837647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.996315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 452.113238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 462.143925] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:27:50 (1773610070) [ 462.274607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.344521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.452337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 462.483974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 470.442463] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:27:58 (1773610078) [ 470.563499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.620445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.745585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.292862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.182522] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:28:07 (1773610087) [ 479.354717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.385648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.598156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.934621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 487.783920] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:28:16 (1773610096) [ 488.077819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.125413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.238263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.579872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.291834] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:28:24 (1773610104) [ 496.536110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.575584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.184566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.466603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.435839] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:28:33 (1773610113) [ 505.756453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.828423] LNet: 20456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.842678] LNet: 20456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 505.869471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.966690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.059679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.901967] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:28:43 (1773610123) [ 515.998955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.053668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.195049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.314644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.706439] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:28:53 (1773610133) [ 525.835562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.903526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.038427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 534.099239] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:29:01 (1773610141) [ 534.511752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.584958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.738808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 542.618971] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:29:10 (1773610150) [ 543.043583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.115725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.236901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 544.042398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 551.978729] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:29:19 (1773610159) [ 552.372781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.496278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.627769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 552.682466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 560.629587] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:29:28 (1773610168) [ 560.726491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.839911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.002442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 561.043745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 569.202163] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:29:37 (1773610177) [ 569.356063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.450607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.609916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 569.677435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 577.491988] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:29:45 (1773610185) [ 577.592482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.671819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.797368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 577.835577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 586.604891] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:29:54 (1773610194) [ 586.721571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.816198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.914603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 587.033455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 587.053334] LNetError: 25775:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 587.100283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 587.151299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 596.064599] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:30:03 (1773610203) [ 596.167282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.218510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.321745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ONvLtBJzSv [ 596.444096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ONvLtBJzSv [ 596.518940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ONvLtBJzSv [ 604.841316] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:30:12 (1773610212) [ 604.991632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.166923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.360891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 605.427140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 605.523657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 614.430596] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:30:22 (1773610222) [ 614.562345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.636417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.800145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 617.466333] LNet: Message forwarding will be enabled in 2s [ 620.579807] LNet: Message forwarding enabled [ 621.155301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 621.184354] LNet: Message forwarding will stop in 2s to 6s [ 625.413116] LNet: Message forwarding disabled [ 625.823409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 627.979256] LNet: Message forwarding will be enabled in 2s [ 631.589050] LNet: Message forwarding enabled [ 636.404101] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:30:44 (1773610244) [ 636.580703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.692520] LNet: 28219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.704873] LNet: 28219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 639.569052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.696076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 647.213648] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:30:55 (1773610255) [ 647.387931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.495917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.629983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 647.677533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 647.729883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 647.804512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 647.864576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 647.929851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 647.970108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 648.017582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 648.067493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 648.130881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 648.178162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 648.228822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 648.265916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 648.316225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 648.369815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 648.419856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 648.533626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 648.628721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 648.700955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 648.749228] LNetError: 28851:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 656.513454] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:31:04 (1773610264) [ 656.642789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.710685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.836732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.121470] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:31:12 (1773610272) [ 667.589189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg415-server [ 669.514407] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:31:17 (1773610277) [ 669.658556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.695901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.781358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.818165] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 671.592390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[16-20]@tcp [ 671.626848] LNet: 30568:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 671.684302] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 671.702983] LNetError: 30358:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.16@tcp has gone from up to down [ 672.675382] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 672.685491] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 673.713834] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 673.727750] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 675.752738] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 675.774542] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 678.980952] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:31:26 (1773610286) [ 679.844382] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 679.872188] LNetError: 30358:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 681.109960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg415-server [ 683.753335] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:31:31 (1773610291) [ 683.803182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.823341] LNetError: 31497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.844088] LNet: Removed LNI 192.168.204.15@tcp [ 684.918750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.071440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.111851] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 686.127000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[16-21/2]@tcp [ 686.161414] LNet: 31622:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 686.179461] LNet: 31622:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 686.211664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[16-21/2]@tcp [ 686.249972] LNetError: 31509:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.16@tcp has gone from up to down [ 686.279131] LNetError: 31509:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 695.868974] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:31:43 (1773610303) [ 695.979882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.034126] LNetError: 32209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.058608] LNet: Removed LNI 192.168.204.15@tcp [ 696.125271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.254902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.325789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.480730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.755551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.850754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.941926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.049519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 697.177760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.297706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.514810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.774892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.858876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.025904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 698.213410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.294935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.437706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.624839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.710953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.808777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 698.970846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.056609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.257115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.481198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.570839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.707923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 699.919351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.061526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.178226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.391632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.492978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.608722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 700.725289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.818752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.051893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.577874] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:31:58 (1773610318) [ 713.181365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg415-server [ 715.864835] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:32:03 (1773610323) [ 719.626640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg415-server [ 723.445211] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:32:09 (1773610329) [ 725.094753] Key type .llcrypt unregistered [ 725.096333] Key type ._llcrypt unregistered [ 735.559308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.675262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.679253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.684514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 738.512789] Key type ._llcrypt registered [ 738.516296] Key type .llcrypt registered [ 738.727720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.742470] alg: No test for adler32 (adler32-zlib) [ 739.817227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.942881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.010394] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 740.978651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.997939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.666755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 742.677728] LNet: Removed LNI 10.1.2.3@tcp [ 742.798404] LNet: 34339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.802861] LNetError: 34339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.813420] LNet: Removed LNI 192.168.204.15@tcp [ 743.262584] Key type .llcrypt unregistered [ 743.264529] Key type ._llcrypt unregistered [ 758.589061] Key type ._llcrypt registered [ 758.593332] Key type .llcrypt registered [ 760.542481] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:32:47 (1773610367) [ 761.175255] Key type .llcrypt unregistered [ 761.177488] Key type ._llcrypt unregistered [ 769.446040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.717955] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.725917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.740388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.334761] Key type ._llcrypt registered [ 772.335796] Key type .llcrypt registered [ 772.488497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.501594] alg: No test for adler32 (adler32-zlib) [ 773.404969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.481683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.500489] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 774.389581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.424736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 777.211179] LNet: 35834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.220257] LNetError: 35834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.225457] LNet: Removed LNI 192.168.204.15@tcp [ 777.234616] LNet: Removed LNI 10.1.2.3@tcp [ 777.959013] Key type .llcrypt unregistered [ 777.962225] Key type ._llcrypt unregistered [ 792.228620] Key type ._llcrypt registered [ 792.231579] Key type .llcrypt registered [ 793.755858] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:33:21 (1773610401) [ 794.326375] Key type .llcrypt unregistered [ 794.328894] Key type ._llcrypt unregistered [ 801.111807] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 801.303445] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 801.310861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 801.325356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.883315] Key type ._llcrypt registered [ 803.885217] Key type .llcrypt registered [ 804.172241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.228281] alg: No test for adler32 (adler32-zlib) [ 805.164861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.237218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 805.262716] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 807.525235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.551948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.093582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 810.116580] LNet: Removed LNI 10.1.2.3@tcp [ 810.139763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 810.195059] LNet: Removed LNI 10.1.2.31@tcp [ 810.211530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 810.237760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.881500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 810.910687] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 811.850988] LNet: 37756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.856831] LNetError: 37756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.862699] LNet: Removed LNI 10.1.2.3@tcp [ 811.881371] LNet: Removed LNI 10.1.2.31@tcp [ 812.679793] Key type .llcrypt unregistered [ 812.682140] Key type ._llcrypt unregistered [ 827.688200] Key type ._llcrypt registered [ 827.691453] Key type .llcrypt registered [ 829.347160] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:33:57 (1773610437) [ 829.739766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.764102] alg: No test for adler32 (adler32-zlib) [ 830.832458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.910670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.939301] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 831.960403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.986282] LNet: 38710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.991619] LNetError: 38710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.004427] LNet: Removed LNI 192.168.204.15@tcp [ 832.037384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.132585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.15@tcp [ 832.165875] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 833.187251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.15@tcp [ 833.230311] LNet: Removed LNI 192.168.204.15@tcp [ 833.337917] LNet: 38845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.086792] Key type .llcrypt unregistered [ 834.091891] Key type ._llcrypt unregistered [ 846.949617] Key type ._llcrypt registered [ 846.950879] Key type .llcrypt registered [ 848.328938] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:34:16 (1773610456) [ 850.009034] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg415-server [ 851.770743] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:34:19 (1773610459) [ 852.448485] Key type .llcrypt unregistered [ 852.452614] Key type ._llcrypt unregistered [ 860.098176] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.341828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.356474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.359306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.186876] Key type ._llcrypt registered [ 863.188678] Key type .llcrypt registered [ 863.453658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.470744] alg: No test for adler32 (adler32-zlib) [ 864.510842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.646869] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 865.400794] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 867.669314] LNet: 40686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.674283] LNetError: 40686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.682550] LNet: Removed LNI 192.168.204.15@tcp1 [ 867.697107] LNet: Removed LNI 10.1.2.3@tcp2 [ 868.242414] Key type .llcrypt unregistered [ 868.244064] Key type ._llcrypt unregistered [ 881.023967] Key type ._llcrypt registered [ 881.027618] Key type .llcrypt registered [ 882.975236] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:34:50 (1773610490) [ 883.453560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.478849] alg: No test for adler32 (adler32-zlib) [ 884.460223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.537549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.569307] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 885.621290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.664576] LNet: 41655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.679533] LNetError: 41655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.695635] LNet: Removed LNI 192.168.204.15@tcp [ 885.721399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.778226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 885.806788] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 893.823206] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:35:02 (1773610502) [ 894.069459] LNet: 42247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.083354] LNetError: 42247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.114474] LNet: Removed LNI 192.168.204.15@tcp [ 894.763812] Key type .llcrypt unregistered [ 894.767915] Key type ._llcrypt unregistered [ 901.546382] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.622790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.634050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.642905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.512546] Key type ._llcrypt registered [ 904.516966] Key type .llcrypt registered [ 904.805555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.837256] alg: No test for adler32 (adler32-zlib) [ 905.852333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.916759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.957676] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 906.712094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 906.727680] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.802770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.823397] LNet: 42993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.831383] LNetError: 42993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.839255] LNet: Removed LNI 192.168.204.15@tcp [ 908.857153] LNet: Removed LNI 10.1.2.3@tcp [ 908.890742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.965671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 908.995840] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 909.003740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.702894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.752617] LNet: 43020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.771729] LNetError: 43020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.784183] LNetError: 43020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 909.786958] LNet: Removed LNI 192.168.204.15@tcp [ 909.833522] LNet: Removed LNI 10.1.2.3@tcp [ 909.865927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.937820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 909.986419] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 909.995307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.504607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 910.523614] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 911.720804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.737721] LNet: 43183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.742658] LNetError: 43183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.747519] LNetError: 43183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 911.750540] LNet: Removed LNI 192.168.204.15@tcp [ 911.752614] LNet: Removed LNI 10.1.2.3@tcp [ 911.761967] LNet: Removed LNI 192.168.204.15@tcp2 [ 911.809300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.879259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 911.902530] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 911.905773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.909387] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 920.501844] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:35:28 (1773610528) [ 920.988335] LNet: 43780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.999844] LNetError: 43780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.009251] LNetError: 43780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 921.017801] LNet: Removed LNI 192.168.204.15@tcp [ 921.019569] LNet: Removed LNI 10.1.2.3@tcp [ 921.042448] LNet: Removed LNI 192.168.204.15@tcp2 [ 921.916769] Key type .llcrypt unregistered [ 921.917744] Key type ._llcrypt unregistered [ 930.985825] Key type ._llcrypt registered [ 930.990430] Key type .llcrypt registered [ 931.329486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.346922] alg: No test for adler32 (adler32-zlib) [ 932.468944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.556041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.586231] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 933.745499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.761793] LNet: 44176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.768860] LNetError: 44176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.789775] LNet: Removed LNI 192.168.204.15@tcp [ 933.833697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.988751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 934.039859] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 934.776497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.815263] LNet: 44206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.821202] LNetError: 44206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.862830] LNet: Removed LNI 192.168.204.15@tcp [ 934.907964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.030403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.071871] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 935.660729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.719202] LNet: Removed LNI 192.168.204.15@tcp [ 935.749137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.843438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.876986] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 936.807183] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 937.251442] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 937.260305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 937.274832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.704725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.718426] LNet: 44315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.722579] LNet: 44315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 939.727736] LNetError: 44315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.731806] LNetError: 44315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 939.748836] LNet: Removed LNI 192.168.204.15@tcp [ 939.799666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.908308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.940324] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 948.740872] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:35:57 (1773610557) [ 949.075448] LNet: 44913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.087009] LNetError: 44913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.118258] LNet: Removed LNI 192.168.204.15@tcp [ 949.868945] Key type .llcrypt unregistered [ 949.876255] Key type ._llcrypt unregistered [ 955.907622] Key type ._llcrypt registered [ 955.914608] Key type .llcrypt registered [ 956.240585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.273553] alg: No test for adler32 (adler32-zlib) [ 957.276105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.352993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.388045] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 958.358063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.381340] LNet: 45312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.387721] LNetError: 45312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.401384] LNet: Removed LNI 192.168.204.15@tcp [ 958.426940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.499892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 958.524713] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 966.483840] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:36:14 (1773610574) [ 966.659991] LNet: 45897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.664521] LNetError: 45897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.683570] LNet: Removed LNI 192.168.204.15@tcp [ 967.202746] Key type .llcrypt unregistered [ 967.204307] Key type ._llcrypt unregistered [ 974.355489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.486983] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.493357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.505332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.204598] Key type ._llcrypt registered [ 976.208334] Key type .llcrypt registered [ 976.452471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.461539] alg: No test for adler32 (adler32-zlib) [ 977.465382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.522784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.547527] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 978.542543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.557320] LNet: 46348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.562497] LNetError: 46348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.573365] LNet: Removed LNI 192.168.204.15@tcp [ 978.594467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.661285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 978.721542] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 988.537792] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:36:35 (1773610595) [ 988.691473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.720797] LNet: 46942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.728142] LNetError: 46942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.742711] LNet: Removed LNI 192.168.204.15@tcp [ 988.770572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.908605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 996.955460] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:36:44 (1773610604) [ 997.288550] LNet: 47531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.059557] Key type .llcrypt unregistered [ 998.063815] Key type ._llcrypt unregistered [ 1006.687634] Key type ._llcrypt registered [ 1006.690452] Key type .llcrypt registered [ 1006.874865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.888037] alg: No test for adler32 (adler32-zlib) [ 1007.820560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.879892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.906285] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1008.820425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.850568] LNet: 47931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.855827] LNetError: 47931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.870292] LNet: Removed LNI 192.168.204.15@tcp [ 1008.906449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.023983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1009.091302] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1016.882796] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:37:04 (1773610624) [ 1017.002789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.040716] LNet: 48517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.055055] LNetError: 48517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.089628] LNet: Removed LNI 192.168.204.15@tcp [ 1017.115743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.232925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1024.162495] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:37:12 (1773610632) [ 1025.911917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1028.129419] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:37:15 (1773610635) [ 1030.241528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1032.137869] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:37:20 (1773610640) [ 1033.850433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1035.604599] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:37:23 (1773610643) [ 1037.498962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1040.043020] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:37:27 (1773610647) [ 1041.858751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1044.522625] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:37:31 (1773610651) [ 1044.661264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.713297] LNet: 50569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.772368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.867406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1044.961773] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1045.974365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.15 [ 1046.004238] LNet: 50710:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1046.018530] LNet: Removed LNI 192.168.204.15@tcp [ 1054.337700] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:37:42 (1773610662) [ 1054.705666] LNet: 51282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.714090] LNetError: 51282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.742304] LNet: Removed LNI 192.168.204.15@tcp [ 1055.548224] Key type .llcrypt unregistered [ 1055.551287] Key type ._llcrypt unregistered [ 1063.730289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.090643] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.097228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1064.108913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.099314] Key type ._llcrypt registered [ 1067.100738] Key type .llcrypt registered [ 1067.324644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.351247] alg: No test for adler32 (adler32-zlib) [ 1068.796649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1077.913605] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:38:05 (1773610685) [ 1078.351384] LNet: 52188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.359395] LNetError: 52188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1078.376194] LNet: Removed LNI 10.1.2.3@tcp [ 1079.424599] Key type .llcrypt unregistered [ 1079.429809] Key type ._llcrypt unregistered [ 1089.051030] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.447273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.459659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.469387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1092.521497] Key type ._llcrypt registered [ 1092.524263] Key type .llcrypt registered [ 1092.734864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.769496] alg: No test for adler32 (adler32-zlib) [ 1093.882447] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1102.522965] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:38:30 (1773610710) [ 1102.660576] LNet: 53079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.671351] LNetError: 53079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1102.709063] LNet: Removed LNI 10.1.2.3@tcp [ 1103.630347] Key type .llcrypt unregistered [ 1103.639702] Key type ._llcrypt unregistered [ 1112.648835] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1112.866323] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1112.870296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1112.874492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1114.617348] Key type ._llcrypt registered [ 1114.622364] Key type .llcrypt registered [ 1114.858959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1114.876425] alg: No test for adler32 (adler32-zlib) [ 1115.908712] LNet: 53376:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1115.925071] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1124.550442] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:38:52 (1773610732) [ 1124.759162] LNet: 53965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.764185] LNetError: 53965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1124.824329] LNet: Removed LNI 10.1.2.3@tcp [ 1125.713428] Key type .llcrypt unregistered [ 1125.714476] Key type ._llcrypt unregistered [ 1133.188843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1133.394969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1133.397603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1133.400854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1136.296943] Key type ._llcrypt registered [ 1136.301289] Key type .llcrypt registered [ 1136.534392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1136.548661] alg: No test for adler32 (adler32-zlib) [ 1137.505763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1137.589399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1148.596484] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:39:16 (1773610756) [ 1149.140079] LNet: 55199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.144596] LNetError: 55199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1149.167231] LNet: Removed LNI 10.1.2.3@tcp [ 1150.114288] Key type .llcrypt unregistered [ 1150.116123] Key type ._llcrypt unregistered [ 1157.512773] Key type ._llcrypt registered [ 1157.520679] Key type .llcrypt registered [ 1157.813857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1157.835384] alg: No test for adler32 (adler32-zlib) [ 1158.878718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1158.929342] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1167.888145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1168.887313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_interrupt -r 1 [ 1168.928354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_interrupt -r 1 [ 1169.003675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1169.042321] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1169.194459] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1169.430449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.468353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1169.510575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.538882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.589564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.631278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.687920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.746472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.774279] LNet: 55637:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.802276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.835461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.861810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.908463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.925744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.947263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.969453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.633294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_dropped -r 1 [ 1170.672870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_dropped -r 1 [ 1170.710632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1170.736044] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1170.944626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.970329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.989575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.012727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.037602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.066670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.089758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.112942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.126415] LNet: 55746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.140705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.161963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.186892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.210813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.241957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.267409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.288788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.842739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_aborted -r 1 [ 1171.868655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_aborted -r 1 [ 1171.893563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1171.909924] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1171.914279] LNet: Skipped 1 previous similar message [ 1172.115594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.147565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.175895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.204788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.229244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.251663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.274331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.294652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.309707] LNet: 55855:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.321694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.342739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.368986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.391647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.418602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.442929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.470776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.898773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_no_route -r 1 [ 1172.928706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_no_route -r 1 [ 1172.957304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1173.199227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.230812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1173.255731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.287798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.320202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.367805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.402155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.435176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.468296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.525108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.563730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.599722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.655407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.713159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.786365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.489788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_timeout -r 1 [ 1174.559711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_timeout -r 1 [ 1174.649100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1174.675990] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1174.680707] LNet: Skipped 2 previous similar messages [ 1175.054114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.094252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1175.146486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.186887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.231653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.266110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.311803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.357965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.390600] LNet: 56074:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.408203] LNet: 56074:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1175.440791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.488830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.548436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.595327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.662532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.716568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.757876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.482985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_error -r 1 [ 1176.520383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_error -r 1 [ 1176.558990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1176.880876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.930773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.990656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.044724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.093256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.142686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.182972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.234941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.315756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.359808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.401237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.443879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.489640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.549440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.649884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.018884] LNet: 56225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.030548] LNetError: 56225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.128666] LNet: Removed LNI 192.168.204.15@tcp [ 1179.874683] Key type .llcrypt unregistered [ 1179.876236] Key type ._llcrypt unregistered [ 1203.317507] Key type ._llcrypt registered [ 1203.323701] Key type .llcrypt registered [ 1205.239897] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:40:12 (1773610812) [ 1206.830715] Key type .llcrypt unregistered [ 1206.846922] Key type ._llcrypt unregistered [ 1215.238484] Key type ._llcrypt registered [ 1215.241777] Key type .llcrypt registered [ 1215.497342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.533390] alg: No test for adler32 (adler32-zlib) [ 1216.552094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.620260] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1226.487855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1240.327238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.400645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1240.451121] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1242.858908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_interrupt -r 1 [ 1242.928678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_interrupt -r 1 [ 1243.001538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_interrupt -r 1 [ 1243.074297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_interrupt -r 1 [ 1243.134638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_interrupt -r 1 [ 1243.247324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_interrupt -r 1 [ 1243.305870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_interrupt -r 1 [ 1243.375703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_interrupt -r 1 [ 1243.440442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1243.471001] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1243.477663] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1243.811243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.859928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1243.897475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.987611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.046125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.098437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.136436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.175200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.202193] LNet: 57695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.237784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.287676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.365728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.392889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.444897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.519811] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1244.536987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.541104] LNet: Skipped 3 previous similar messages [ 1244.660645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.476391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_dropped -r 1 [ 1245.526884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_dropped -r 1 [ 1245.577633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_dropped -r 1 [ 1245.624436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_dropped -r 1 [ 1245.698566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_dropped -r 1 [ 1245.761704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_dropped -r 1 [ 1245.853546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_dropped -r 1 [ 1245.960892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_dropped -r 1 [ 1246.063342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1246.118249] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1246.536693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.571258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.597359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.663499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.759154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.807375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.855538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.891305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.931671] LNet: 57817:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.943941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.970632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.005489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.035142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.080352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.109346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.149256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.715364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_aborted -r 1 [ 1247.792538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_aborted -r 1 [ 1247.834166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_aborted -r 1 [ 1247.880679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_aborted -r 1 [ 1247.916241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_aborted -r 1 [ 1247.952987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_aborted -r 1 [ 1248.010545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_aborted -r 1 [ 1248.060646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_aborted -r 1 [ 1248.098608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1248.418343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.459838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1248.529769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.581247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.610407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.650836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.683547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.738394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.759771] LNet: 57938:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.780642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.813313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.847966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.869522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.892387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.918841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.943780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1249.486567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_no_route -r 1 [ 1249.519331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_no_route -r 1 [ 1249.550182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_no_route -r 1 [ 1249.578892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_no_route -r 1 [ 1249.612280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_no_route -r 1 [ 1249.650854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_no_route -r 1 [ 1249.690883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_no_route -r 1 [ 1249.717728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_no_route -r 1 [ 1249.762558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1249.789942] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1249.793613] LNet: Skipped 5 previous similar messages [ 1250.121685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.172651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1250.205339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.233913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.266810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.291993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.323808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.353303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.379468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.418732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.498245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.546736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.586800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.634849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.681410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1251.355205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_timeout -r 1 [ 1251.431354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_timeout -r 1 [ 1251.500825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_timeout -r 1 [ 1251.568192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_timeout -r 1 [ 1251.626371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_timeout -r 1 [ 1251.705901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_timeout -r 1 [ 1251.757879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_timeout -r 1 [ 1251.830732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_timeout -r 1 [ 1251.897689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1252.246462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.283479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1252.310334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.346594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.385501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.413769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.444494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.488422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.507452] LNet: 58180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.513213] LNet: 58180:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1252.551770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.618889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.679570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.729804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.768993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.796138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.830365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.488303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_error -r 1 [ 1253.523626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_error -r 1 [ 1253.591177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_error -r 1 [ 1253.644616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_error -r 1 [ 1253.727223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_error -r 1 [ 1253.806833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_error -r 1 [ 1253.860880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_error -r 1 [ 1253.907422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_error -r 1 [ 1253.969414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1254.006706] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1254.014987] LNet: Skipped 5 previous similar messages [ 1254.358875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.415118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1254.447851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.495862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.523786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.560610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.602823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.639477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.690645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.750268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.830290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.872217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.930266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.980252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.074813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1261.605534] LNet: 58378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.612643] LNetError: 58378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.624197] LNet: Removed LNI 192.168.204.15@tcp [ 1262.630523] LNetError: 58378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.647474] LNet: Removed LNI 192.168.204.15@tcp1 [ 1263.197848] Key type .llcrypt unregistered [ 1263.204990] Key type ._llcrypt unregistered [ 1286.330424] Key type ._llcrypt registered [ 1286.332574] Key type .llcrypt registered [ 1288.289371] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:41:35 (1773610895) [ 1289.493824] Key type .llcrypt unregistered [ 1289.498114] Key type ._llcrypt unregistered [ 1296.689593] Key type ._llcrypt registered [ 1296.692318] Key type .llcrypt registered [ 1296.873816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.886379] alg: No test for adler32 (adler32-zlib) [ 1297.791179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1297.840899] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1307.597224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1308.777957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_dropped -r 1 [ 1308.879321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_dropped -r 1 [ 1308.953880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1309.001937] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1309.300507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.345679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1309.379227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.411334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.449950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.509182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.551571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.598592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.622811] LNet: 59631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.644305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1309.696726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1309.743953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.803596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.867346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.942492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.977100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1310.544308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_error -r 1 [ 1310.572256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_error -r 1 [ 1310.598776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1310.617514] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1310.912804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1310.952186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1310.995873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.098928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.142620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.195167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.242451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.302358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.368779] LNet: 59741:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.413386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1311.475282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1311.514446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1311.599369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.654346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1311.721362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1311.755248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1312.575909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_timeout -r 1 [ 1312.636447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_timeout -r 1 [ 1312.678203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1312.707767] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1313.090636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1313.146776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1313.194780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1313.240336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1313.277100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.324357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.370675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.415431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.448238] LNet: 59850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1313.462128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.501603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.544304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.591945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.622549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.665640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1313.733041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1314.092661] LNet: 59891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.104437] LNetError: 59891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.125276] LNet: Removed LNI 192.168.204.15@tcp [ 1314.868715] Key type .llcrypt unregistered [ 1314.876606] Key type ._llcrypt unregistered [ 1338.217922] Key type ._llcrypt registered [ 1338.220213] Key type .llcrypt registered [ 1339.887556] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:42:27 (1773610947) [ 1340.937433] Key type .llcrypt unregistered [ 1340.949270] Key type ._llcrypt unregistered [ 1348.907941] Key type ._llcrypt registered [ 1348.910744] Key type .llcrypt registered [ 1349.089904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.101550] alg: No test for adler32 (adler32-zlib) [ 1350.085983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1350.145673] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1359.130394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1372.722728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.795066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1372.861700] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1374.596897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_dropped -r 1 [ 1374.666614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e remote_dropped -r 1 [ 1374.721685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_dropped -r 1 [ 1374.778326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e remote_dropped -r 1 [ 1374.831575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e remote_dropped -r 1 [ 1374.888509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e remote_dropped -r 1 [ 1374.958078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e remote_dropped -r 1 [ 1375.048772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e remote_dropped -r 1 [ 1375.107092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1375.142284] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1375.153511] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 1375.411082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1375.436123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1375.458321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1375.498870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1375.524415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1375.551232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1375.579156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1375.602778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1375.615258] LNet: 61379:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1375.632395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1375.654839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1375.676948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.699692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1375.721557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.755924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.783560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1375.809932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.463731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_error -r 1 [ 1376.550338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e remote_error -r 1 [ 1376.616726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_error -r 1 [ 1376.664332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e remote_error -r 1 [ 1376.704438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e remote_error -r 1 [ 1376.748437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e remote_error -r 1 [ 1376.812652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e remote_error -r 1 [ 1376.867700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e remote_error -r 1 [ 1376.933645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1376.971702] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1376.978428] LNet: Skipped 2 previous similar messages [ 1377.271909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1377.309828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1377.367265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1377.470944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1377.519423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1377.570028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1377.612907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1377.662431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1377.702807] LNet: 61502:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.731437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1377.778614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1377.828978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.882340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.959771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.024555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1378.059485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.093343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.784783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_timeout -r 1 [ 1378.829394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e remote_timeout -r 1 [ 1378.880438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_timeout -r 1 [ 1378.951745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e remote_timeout -r 1 [ 1379.009674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e remote_timeout -r 1 [ 1379.063682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e remote_timeout -r 1 [ 1379.105404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e remote_timeout -r 1 [ 1379.152294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e remote_timeout -r 1 [ 1379.196971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.115@tcp [ 1379.226266] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1379.239746] LNet: Skipped 1 previous similar message [ 1379.570380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1379.622662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1379.665503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.730342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.801080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.837370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.896978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.964432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.996876] LNet: 61626:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1380.041979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1380.089591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1380.158713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.221990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1380.285467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.326993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.404966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1380.449768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.820862] LNet: 61703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.828764] LNetError: 61703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.852452] LNet: Removed LNI 192.168.204.15@tcp [ 1386.861090] LNetError: 61703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.894364] LNet: Removed LNI 192.168.204.15@tcp1 [ 1387.534904] Key type .llcrypt unregistered [ 1387.538697] Key type ._llcrypt unregistered [ 1407.560339] Key type ._llcrypt registered [ 1407.561994] Key type .llcrypt registered [ 1408.895195] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:43:36 (1773611016) [ 1409.464562] Key type .llcrypt unregistered [ 1409.468605] Key type ._llcrypt unregistered [ 1416.357094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1416.732304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1416.735206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1416.738124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1419.634205] Key type ._llcrypt registered [ 1419.636709] Key type .llcrypt registered [ 1419.866689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.880549] alg: No test for adler32 (adler32-zlib) [ 1420.878228] LNet: 62785:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1420.943602] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1421.025568] LNet: 62799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.045650] LNetError: 62799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.071903] LNet: Removed LNI 192.168.204.15@tcp [ 1421.904913] Key type .llcrypt unregistered [ 1421.916235] Key type ._llcrypt unregistered [ 1429.834124] Key type ._llcrypt registered [ 1429.835870] Key type .llcrypt registered [ 1430.048718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.072684] alg: No test for adler32 (adler32-zlib) [ 1431.087700] LNet: 63059:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1431.121293] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1431.123435] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1431.185258] LNet: 63073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.198090] LNetError: 63073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.201033] LNet: Removed LNI 192.168.204.15@tcp [ 1431.228947] LNet: Removed LNI 10.1.2.3@tcp1 [ 1432.047598] Key type .llcrypt unregistered [ 1432.054441] Key type ._llcrypt unregistered [ 1440.198954] Key type ._llcrypt registered [ 1440.203376] Key type .llcrypt registered [ 1440.468846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1440.488241] alg: No test for adler32 (adler32-zlib) [ 1441.491250] LNet: 63333:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1441.505036] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1441.511882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1441.563050] LNet: 63347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.569960] LNetError: 63347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.576971] LNet: Removed LNI 192.168.204.15@tcp [ 1441.589315] LNet: Removed LNI 10.1.2.3@tcp [ 1442.443385] Key type .llcrypt unregistered [ 1442.449195] Key type ._llcrypt unregistered [ 1450.058989] Key type ._llcrypt registered [ 1450.062269] Key type .llcrypt registered [ 1450.452952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.493069] alg: No test for adler32 (adler32-zlib) [ 1451.493405] LNet: 63608:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1451.504377] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1451.507915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1451.538505] LNet: 63622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.543816] LNetError: 63622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.548539] LNet: Removed LNI 192.168.204.15@tcp [ 1451.555419] LNet: Removed LNI 10.1.2.3@tcp [ 1452.540590] Key type .llcrypt unregistered [ 1452.553409] Key type ._llcrypt unregistered [ 1460.793649] Key type ._llcrypt registered [ 1460.796793] Key type .llcrypt registered [ 1461.030301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1461.067516] alg: No test for adler32 (adler32-zlib) [ 1462.113744] LNet: 63887:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1462.152109] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1462.158373] LNetError: 63887:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1462.167978] LNetError: Error -100 starting up LNI tcp [ 1462.188642] LNet: Removed LNI 192.168.204.15@tcp [ 1463.391850] Key type .llcrypt unregistered [ 1463.394337] Key type ._llcrypt unregistered [ 1478.613693] Key type ._llcrypt registered [ 1478.615378] Key type .llcrypt registered [ 1480.242830] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:44:47 (1773611087) [ 1481.401551] Key type .llcrypt unregistered [ 1481.404517] Key type ._llcrypt unregistered [ 1490.569915] Key type ._llcrypt registered [ 1490.572964] Key type .llcrypt registered [ 1490.850928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.871737] alg: No test for adler32 (adler32-zlib) [ 1491.918637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1491.970445] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1501.344814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1502.332887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e network_timeout -r 1 [ 1502.394089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e network_timeout -r 1 [ 1502.465276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1502.506994] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1502.825850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1502.854859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1502.913686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1502.942384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1502.993074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1503.039146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1503.078436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1503.135432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1503.172479] LNet: 65146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1503.212475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1503.258537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1503.272323] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1503.281736] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1503.305501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1503.349497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1503.379769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1503.433655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1503.725123] LNet: 65185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.728769] LNetError: 65185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.742389] LNet: Removed LNI 192.168.204.15@tcp [ 1504.210683] Key type .llcrypt unregistered [ 1504.212403] Key type ._llcrypt unregistered [ 1529.889423] Key type ._llcrypt registered [ 1529.897778] Key type .llcrypt registered [ 1530.129807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1530.146568] alg: No test for adler32 (adler32-zlib) [ 1531.228219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1531.266062] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1541.947532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1554.702810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.764645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.795822] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1556.819347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e network_timeout -r 1 [ 1556.860380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e network_timeout -r 1 [ 1556.901370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e network_timeout -r 1 [ 1556.949667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e network_timeout -r 1 [ 1557.007662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e network_timeout -r 1 [ 1557.067419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e network_timeout -r 1 [ 1557.140763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e network_timeout -r 1 [ 1557.204939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e network_timeout -r 1 [ 1557.280821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 1557.329397] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1557.743505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1557.785579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1557.830896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1557.866913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1557.915620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1557.920380] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1557.941804] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 1558.012476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1558.084305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1558.137428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1558.188506] LNet: 66069:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.220501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1558.280983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1558.342046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1558.396469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1558.434780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1558.520546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1563.579457] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1564.036272] LNet: 66143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.048314] LNetError: 66143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.055230] LNet: Removed LNI 192.168.204.15@tcp [ 1564.088074] LNet: Removed LNI 192.168.204.15@tcp1 [ 1564.651703] Key type .llcrypt unregistered [ 1564.655819] Key type ._llcrypt unregistered [ 1586.439549] Key type ._llcrypt registered [ 1586.441121] Key type .llcrypt registered [ 1588.519378] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:46:35 (1773611195) [ 1589.186683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.211270] alg: No test for adler32 (adler32-zlib) [ 1590.235093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.368849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.417773] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1592.077682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1592.134965] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1593.617646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 1593.692714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1593.738813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1593.788586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1593.869667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1593.908335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1593.939780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.15@tcp [ 1593.957333] LNetError: 67246:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.15@tcp to recovery [ 1594.400495] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1594.400775] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1596.448630] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1597.794633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1600.544387] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1601.456923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1601.478558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1601.525717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.553702] LNet: 67353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.558577] LNetError: 67353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.562214] LNet: Removed LNI 192.168.204.15@tcp [ 1601.574327] LNet: Removed LNI 192.168.204.15@tcp1 [ 1601.597348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.654653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.671325] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1602.093686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1602.128129] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1602.833721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 1602.855235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1602.901057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1602.938556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1602.988521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1603.056294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1603.131697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1603.174708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.15@tcp [ 1603.190955] LNetError: 67625:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.15@tcp to recovery [ 1603.684063] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1607.006495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1609.825671] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1609.837944] LNet: Skipped 1 previous similar message [ 1614.253325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1614.331914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1614.388804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.274629] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:47:11 (1773611231) [ 1623.557830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.591105] LNet: 68331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.610411] LNetError: 68331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.621090] LNetError: 68331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1623.630690] LNet: Removed LNI 192.168.204.15@tcp [ 1623.647467] LNet: Removed LNI 192.168.204.15@tcp1 [ 1623.667350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.724305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.747203] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1624.681777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.741727] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1625.797753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 1625.901747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1625.945051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1626.004425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1626.053916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1626.092397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1626.127253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.15@tcp --health 0 [ 1626.228804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1626.784236] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.15@tcp [ 1627.809146] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1627.818909] LNet: Skipped 1 previous similar message [ 1641.886977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1642.100884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1642.147179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.188445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.15@tcp --health 500 [ 1642.290451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.168803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.245354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.273077] LNet: 68827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.277862] LNetError: 68827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.288820] LNetError: 68827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1649.308929] LNet: Removed LNI 192.168.204.15@tcp [ 1649.323543] LNet: Removed LNI 192.168.204.15@tcp1 [ 1649.358310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.424950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.457270] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1650.191347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.230000] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1651.343525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 1651.380641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1651.484367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1651.525702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1651.566863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1651.647884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1651.712971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1651.789943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.15@tcp --health 0 [ 1652.452243] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1652.466106] LNet: Skipped 3 previous similar messages [ 1653.250477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1663.730736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1663.778178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1663.834532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1672.065757] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:48:00 (1773611280) [ 1672.670116] LNet: 69874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.674928] LNetError: 69874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.683178] LNetError: 69874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1672.692401] LNet: Removed LNI 192.168.204.15@tcp [ 1672.702825] LNet: Removed LNI 192.168.204.15@tcp1 [ 1673.259870] Key type .llcrypt unregistered [ 1673.261337] Key type ._llcrypt unregistered [ 1680.426940] Key type ._llcrypt registered [ 1680.428399] Key type .llcrypt registered [ 1680.813529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.834950] alg: No test for adler32 (adler32-zlib) [ 1681.872795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1681.910334] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1692.728896] Lustre: DEBUG MARKER: Initial discovery [ 1692.798029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.115@tcp [ 1696.086257] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1696.258596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1696.318743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.115@tcp [ 1696.358893] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1696.413748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1696.472816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.115@tcp -r 1 -m GET -l 3 [ 1696.505213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.115@tcp -r 1 -m GET -e local_error [ 1696.550411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.115@tcp -r 1 -m PUT -l 6 [ 1697.316558] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1698.891987] Lustre: DEBUG MARKER: Force oleg415-client.virtnet to discover 192.168.204.115@tcp (in background) [ 1699.061516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.115@tcp [ 1701.172006] Lustre: DEBUG MARKER: Force oleg415-server to discover 192.168.204.15@tcp [ 1702.176719] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1705.728814] Lustre: DEBUG MARKER: Wait for 70629 [ 1705.962853] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 1708.099545] Lustre: DEBUG MARKER: Finished wait on 70629 [ 1708.303060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1708.398752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1708.907596] LNet: 71089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.916463] LNetError: 71089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.994188] LNet: Removed LNI 192.168.204.15@tcp [ 1710.881156] Key type .llcrypt unregistered [ 1710.883430] Key type ._llcrypt unregistered [ 1733.287409] Key type ._llcrypt registered [ 1733.290292] Key type .llcrypt registered [ 1734.672723] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:49:02 (1773611342) [ 1735.306663] Key type .llcrypt unregistered [ 1735.310168] Key type ._llcrypt unregistered [ 1742.848484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1743.114530] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1743.116591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1743.138830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1745.837968] Key type ._llcrypt registered [ 1745.840416] Key type .llcrypt registered [ 1746.066380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1746.083860] alg: No test for adler32 (adler32-zlib) [ 1747.131688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.222177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.268803] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1748.330416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1748.358478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1758.578609] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:49:26 (1773611366) [ 1759.023063] LNet: 73170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.033903] LNetError: 73170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.046295] LNet: Removed LNI 192.168.204.15@tcp [ 1759.071636] LNet: Removed LNI 10.1.2.3@tcp [ 1759.871541] Key type .llcrypt unregistered [ 1759.873892] Key type ._llcrypt unregistered [ 1768.911762] Key type ._llcrypt registered [ 1768.914478] Key type .llcrypt registered [ 1769.234625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1769.263698] alg: No test for adler32 (adler32-zlib) [ 1770.273493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.339563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.371820] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1771.407359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1771.446416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1772.459985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.15@tcp [ 1773.536340] LNetError: Unexpected error -22 connecting to 192.168.204.15@tcp at host 192.168.204.15:988 [ 1780.749928] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:49:48 (1773611388) [ 1781.120052] LNet: 74302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.133952] LNetError: 74302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.155726] LNet: Removed LNI 192.168.204.15@tcp [ 1781.189879] LNet: Removed LNI 10.1.2.3@tcp [ 1781.926538] Key type .llcrypt unregistered [ 1781.932143] Key type ._llcrypt unregistered [ 1790.204028] Key type ._llcrypt registered [ 1790.206140] Key type .llcrypt registered [ 1790.405476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1790.420989] alg: No test for adler32 (adler32-zlib) [ 1791.460530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.577902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1791.602267] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1792.565551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1792.584321] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 1793.725525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.15@tcp1 --nid 192.168.204.15@tcp2 [ 1798.267810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp1 192.168.204.15@tcp2 [ 1798.283393] LNetError: 75140:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.15@tcp1 to 192.168.204.15@tcp2 - Route table may be misconfigured [ 1798.303734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp2 192.168.204.15@tcp1 [ 1805.452308] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:50:13 (1773611413) [ 1805.628935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.638284] LNet: 75714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.641599] LNetError: 75714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.645342] LNet: Removed LNI 192.168.204.15@tcp1 [ 1806.698063] LNet: Removed LNI 192.168.204.15@tcp2 [ 1806.776498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.911972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.945940] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1807.591894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1807.610102] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1808.455448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 1808.536956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.15@tcp [ 1808.581670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp1 192.168.204.15@tcp1 [ 1808.623923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.15@tcp -d 192.168.204.15@tcp -e network_timeout [ 1808.667756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -e network_timeout [ 1808.738527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -e network_timeout [ 1808.774842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -e network_timeout [ 1808.793652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1808.813831] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 1808.848899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1808.864522] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 1808.993559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1809.105407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1816.049673] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:50:24 (1773611424) [ 1816.561285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.606916] LNet: 76576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.622709] LNetError: 76576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.636241] LNetError: 76576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1816.648084] LNet: Removed LNI 192.168.204.15@tcp [ 1817.712538] LNet: Removed LNI 192.168.204.15@tcp1 [ 1817.803825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.938544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1818.276396] LNet: 76614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.820434] Key type .llcrypt unregistered [ 1818.822936] Key type ._llcrypt unregistered [ 1832.936331] Key type ._llcrypt registered [ 1832.939133] Key type .llcrypt registered [ 1834.517284] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:50:42 (1773611442) [ 1835.004374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.023269] alg: No test for adler32 (adler32-zlib) [ 1836.016197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.764512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1840.004478] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:50:47 (1773611447) [ 1840.131059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.168452] LNet: 77709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.207790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.308694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1840.367056] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1841.169203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1841.199495] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 1842.123194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1842.155448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp1 [ 1842.189287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp1 [ 1849.693134] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:50:57 (1773611457) [ 1852.005986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg415-server [ 1853.976521] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:51:02 (1773611462) [ 1856.512864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg415-server [ 1859.125963] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:51:06 (1773611466) [ 1861.774868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg415-server [ 1864.040667] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:51:11 (1773611471) [ 1866.891663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg415-server [ 1869.000650] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:51:16 (1773611476) [ 1871.462356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg415-server [ 1874.344669] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:51:21 (1773611481) [ 1876.481157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg415-server [ 1879.609630] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:51:26 (1773611486) [ 1883.955880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg415-server [ 1887.464790] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:51:34 (1773611494) [ 1889.702813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg415-server [ 1891.950120] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:51:40 (1773611500) [ 1894.461828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg415-server [ 1897.154968] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:51:44 (1773611504) [ 1897.245778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.276698] LNet: 81586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.285504] LNetError: 81586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.291242] LNet: Removed LNI 192.168.204.15@tcp [ 1898.350064] LNet: Removed LNI 192.168.204.15@tcp1 [ 1898.417307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.501615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.621659] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1899.589725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1899.695916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1900.011946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.044414] LNet: 81741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.051264] LNetError: 81741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.055490] LNetError: 81741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1900.068289] LNet: Removed LNI 192.168.204.15@tcp [ 1900.084186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.138201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1900.176398] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1901.471035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1901.700218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.741105] LNetError: 81886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.766394] LNet: Removed LNI 192.168.204.15@tcp [ 1902.871995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.010261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1903.077162] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1904.346923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1904.438668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1904.631421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.654537] LNet: 82034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.662200] LNet: 82034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1904.668611] LNetError: 82034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.688453] LNet: Removed LNI 192.168.204.15@tcp [ 1904.702403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.766983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1904.793851] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1906.049148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1907.463126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.520490] LNet: Removed LNI 192.168.204.15@tcp [ 1908.577671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.684742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.720865] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1909.612643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1909.743623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1910.017036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.030455] LNet: 82336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.033631] LNet: 82336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1910.040877] LNetError: 82336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.044941] LNetError: 82336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1911.083533] LNet: Removed LNI 192.168.204.15@tcp [ 1911.110660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.197081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1911.230647] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1912.057766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1912.348884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.447428] LNet: Removed LNI 192.168.204.15@tcp [ 1913.460698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.531831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.590492] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1914.211894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1914.333333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1914.625432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.689325] LNet: Removed LNI 192.168.204.15@tcp [ 1915.714858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.817760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1915.835080] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1916.688158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1916.940538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.992598] LNet: Removed LNI 192.168.204.15@tcp [ 1918.043774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.140936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.189779] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1919.023736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1919.187772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1919.659145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.688466] LNet: 82921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.702450] LNet: 82921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1919.711635] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.717100] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1920.748424] LNet: Removed LNI 192.168.204.15@tcp [ 1920.797409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.921257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1920.987266] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1921.943453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1922.309862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.372775] LNet: Removed LNI 192.168.204.15@tcp [ 1923.418455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.498775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1923.533888] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1924.553315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1924.727726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1925.240569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1926.313669] LNet: Removed LNI 192.168.204.15@tcp [ 1926.397695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.551469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1926.608865] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1927.491903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 1927.790135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.872952] LNet: Removed LNI 192.168.204.15@tcp [ 1928.896461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.971767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.037886] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1930.363495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1930.445988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.475695] LNet: Removed LNI 192.168.204.15@tcp [ 1930.496637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.554377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.608591] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1932.068909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1943.714320] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:52:30 (1773611550) [ 1944.001577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.084214] LNet: 84192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.108755] LNet: 84192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1944.137324] LNetError: 84192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.163769] LNetError: 84192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1944.213338] LNet: Removed LNI 192.168.204.15@tcp [ 1944.293839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.418521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1944.456251] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/180] [ 1947.469599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.585315] LNet: Removed LNI 192.168.204.15@tcp231 [ 1947.675698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.803390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1947.866454] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/180] [ 1947.878043] LNet: 84346:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1948.657145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1948.708462] LNet: Removed LNI 192.168.204.15@tcp231 [ 1948.729431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1948.782762] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/180] [ 1950.501572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.517353] LNet: Removed LNI 192.168.204.15@tcp231 [ 1950.535335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.581550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1950.600083] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/0] [ 1950.604288] LNet: 84498:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1950.846573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1950.874512] LNet: Removed LNI 192.168.204.15@tcp231 [ 1950.894163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1950.918212] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/0] [ 1951.876121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1951.939548] LNet: Removed LNI 192.168.204.15@tcp231 [ 1951.996636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.062225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1952.094549] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/60] [ 1952.102470] LNet: 84650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1952.401577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1952.440513] LNet: Removed LNI 192.168.204.15@tcp231 [ 1952.470578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1952.508151] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/60] [ 1953.732227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.786561] LNet: Removed LNI 192.168.204.15@tcp231 [ 1953.812417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.882044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1953.913494] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/180] [ 1954.653358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1954.712977] LNet: Removed LNI 192.168.204.15@tcp231 [ 1954.743510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1954.795035] LNet: Added LNI 192.168.204.15@tcp231 [8/256/0/180] [ 1964.884384] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:52:53 (1773611573) [ 1965.050350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.136379] LNet: Removed LNI 192.168.204.15@tcp231 [ 1965.187758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.288688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1965.315734] LNet: Added LNI 192.168.204.15@tcp232 [8/256/0/180] [ 1965.951640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1965.985990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1966.015522] LNet: Removed LNI 192.168.204.15@tcp232 [ 1966.029472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1966.042354] LNet: Added LNI 192.168.204.15@tcp232 [8/256/0/180] [ 1966.655965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1966.688392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1966.712670] LNet: Removed LNI 192.168.204.15@tcp232 [ 1966.726901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1966.743712] LNet: Added LNI 192.168.204.15@tcp232 [8/256/0/180] [ 1967.674501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1967.714609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1967.740966] LNet: Removed LNI 192.168.204.15@tcp232 [ 1967.755180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1967.782362] LNet: Added LNI 192.168.204.15@tcp232 [8/256/0/180] [ 1968.825150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1968.959650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1969.022264] LNet: Removed LNI 192.168.204.15@tcp232 [ 1969.054904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1976.717193] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:53:04 (1773611584) [ 1977.531150] LNet: 86641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.541450] LNet: 86641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1978.226954] Key type .llcrypt unregistered [ 1978.230459] Key type ._llcrypt unregistered [ 1986.370783] Key type ._llcrypt registered [ 1986.375620] Key type .llcrypt registered [ 1986.625932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.659258] alg: No test for adler32 (adler32-zlib) [ 1987.628768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1987.659818] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 1996.156989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 2006.553066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.616727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2006.665847] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 2008.620387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_interrupt -r 2 [ 2008.664280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_interrupt -r 2 [ 2008.717452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_interrupt -r 2 [ 2008.751861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_interrupt -r 2 [ 2008.815790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_interrupt -r 2 [ 2008.862811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_interrupt -r 2 [ 2008.948846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_interrupt -r 2 [ 2008.995217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_interrupt -r 2 [ 2009.058152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2009.086614] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 2009.114368] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 2009.506500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2009.540756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2009.574383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2009.613895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2009.646519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2009.673854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2009.699087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2009.728333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2009.756371] LNet: 87305:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2009.771519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2009.814154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2009.896577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2009.940613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2009.981425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2010.009923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2010.663269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_dropped -r 2 [ 2010.693959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_dropped -r 2 [ 2010.721199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_dropped -r 2 [ 2010.788510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_dropped -r 2 [ 2010.822859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_dropped -r 2 [ 2010.863518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_dropped -r 2 [ 2010.932177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_dropped -r 2 [ 2010.986565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_dropped -r 2 [ 2011.031557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2011.058195] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 2011.501474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2011.563549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2011.618415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2011.694766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2011.739699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2011.774326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2011.852620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2011.897876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2011.931717] LNet: 87425:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2011.973663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2012.005420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2012.116111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2012.189913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2012.250689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2012.286567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2012.931627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_aborted -r 2 [ 2012.978128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_aborted -r 2 [ 2013.019304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_aborted -r 2 [ 2013.073953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_aborted -r 2 [ 2013.126307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_aborted -r 2 [ 2013.168468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_aborted -r 2 [ 2013.217804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_aborted -r 2 [ 2013.257649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_aborted -r 2 [ 2013.341948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2013.382285] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 2013.668505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2013.698395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2013.727595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2013.769284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2013.804888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2013.839613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2013.881890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2013.915880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2013.933341] LNet: 87546:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2013.952700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2013.993131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2014.035268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2014.068365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2014.100873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2014.130670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2014.792484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_no_route -r 2 [ 2014.860567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_no_route -r 2 [ 2014.908161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_no_route -r 2 [ 2014.959130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_no_route -r 2 [ 2015.002648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_no_route -r 2 [ 2015.056885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_no_route -r 2 [ 2015.109388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_no_route -r 2 [ 2015.171058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_no_route -r 2 [ 2015.242325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2015.508811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2015.566628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2015.596719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2015.635732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2015.662158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2015.697288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2015.753100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2015.799136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2015.829876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2015.875577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2015.910401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2015.943753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2015.972600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2015.997965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2016.616528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e local_timeout -r 2 [ 2016.657782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e local_timeout -r 2 [ 2016.697579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e local_timeout -r 2 [ 2016.735967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e local_timeout -r 2 [ 2016.795059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e local_timeout -r 2 [ 2016.870598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e local_timeout -r 2 [ 2016.942957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e local_timeout -r 2 [ 2017.041327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e local_timeout -r 2 [ 2017.091742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2017.108764] LNet: There was an unexpected network error while writing to 192.168.204.115: rc = -22 [ 2017.115943] LNet: Skipped 1 previous similar message [ 2017.347483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2017.392949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2017.459880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2017.513527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2017.548280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2017.593231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.650701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.710558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.730967] LNet: 87786:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.749209] LNet: 87786:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2017.774812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.822436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.864967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2017.908982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2017.966764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.008709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.738992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp -m GET -e remote_dropped -r 2 [ 2018.784861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp -d 192.168.204.115@tcp1 -m GET -e remote_dropped -r 2 [ 2018.840107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp -m GET -e remote_dropped -r 2 [ 2018.878824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp -d 192.168.204.15@tcp1 -m GET -e remote_dropped -r 2 [ 2018.911053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp -m GET -e remote_dropped -r 2 [ 2018.975592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.115@tcp1 -m GET -e remote_dropped -r 2 [ 2019.014058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp -m GET -e remote_dropped -r 2 [ 2019.066411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.15@tcp1 -d 192.168.204.15@tcp1 -m GET -e remote_dropped -r 2 [ 2019.130558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.15@tcp 192.168.204.115@tcp [ 2019.160073] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 2019.449686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2019.485839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2019.522777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2019.558864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2019.585365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2019.614730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2019.643918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2019.675684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2019.701900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2019.734796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2019.765501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2019.785986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2019.819422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2019.860228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2026.696749] LNet: 87981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.704984] LNetError: 87981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.707777] LNet: Removed LNI 192.168.204.15@tcp [ 2026.736810] LNet: Removed LNI 192.168.204.15@tcp1 [ 2027.220962] Key type .llcrypt unregistered [ 2027.225121] Key type ._llcrypt unregistered [ 2045.139357] Key type ._llcrypt registered [ 2045.146839] Key type .llcrypt registered [ 2046.754350] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:54:14 (1773611654) [ 2047.113485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.138360] alg: No test for adler32 (adler32-zlib) [ 2048.095909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.199995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2048.220946] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2049.004587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2049.040456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.15@tcp [ 2049.075093] LNetError: 88962:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.15@tcp to recovery [ 2049.195265] LNet: 1 local NIs in recovery (showing 1): 192.168.204.15@tcp [ 2049.230333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.15@tcp [ 2049.380889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2057.763592] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:54:25 (1773611665) [ 2057.897582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.929937] LNet: 89543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.936941] LNetError: 89543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.948261] LNet: Removed LNI 192.168.204.15@tcp [ 2057.968181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.051272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.099139] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2059.261859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2059.293162] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 2060.677201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.15@tcp [ 2060.699263] LNetError: 89818:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.204.15@tcp to recovery [ 2061.124301] LNet: There was an unexpected network error while writing to 192.168.204.15: rc = -22 [ 2062.002741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.15@tcp [ 2070.538221] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:54:38 (1773611678) [ 2072.696737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg415-server [ 2074.679328] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:54:42 (1773611682) [ 2076.585944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg415-server [ 2079.044773] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:54:46 (1773611686) [ 2081.200491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg415-server [ 2083.120757] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:54:51 (1773611691) [ 2083.568863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.596705] LNet: 91441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.611094] LNetError: 91441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.619672] LNet: Removed LNI 192.168.204.15@tcp [ 2083.646223] LNet: Removed LNI 192.168.204.15@tcp1 [ 2083.689820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.774807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.834177] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2084.933538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2085.065415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2085.126865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2085.234271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2085.346339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2085.426888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2092.530312] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:55:00 (1773611700) [ 2092.926775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.961401] LNet: 92166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.966865] LNetError: 92166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.974950] LNetError: 92166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2092.997250] LNet: Removed LNI 192.168.204.15@tcp [ 2093.035803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.129679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2093.160107] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2102.143161] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:55:10 (1773611710) [ 2102.464093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.494982] LNet: 92871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.501088] LNetError: 92871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.520110] LNet: Removed LNI 192.168.204.15@tcp [ 2102.565917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.713197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.738745] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2110.953177] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:55:19 (1773611719) [ 2112.470767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2114.127979] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:55:22 (1773611722) [ 2114.921869] LNet: 93906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.926984] LNetError: 93906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.940480] LNet: Removed LNI 192.168.204.15@tcp [ 2115.593045] Key type .llcrypt unregistered [ 2115.594863] Key type ._llcrypt unregistered [ 2123.433319] Key type ._llcrypt registered [ 2123.435688] Key type .llcrypt registered [ 2123.691989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2123.703545] alg: No test for adler32 (adler32-zlib) [ 2124.621585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2124.649993] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2132.495704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 2137.658183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.115@tcp [ 2138.145555] LNet: 94324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.161037] LNetError: 94324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.211411] LNet: Removed LNI 192.168.204.15@tcp [ 2138.830781] Key type .llcrypt unregistered [ 2138.837108] Key type ._llcrypt unregistered [ 2152.555712] Key type ._llcrypt registered [ 2152.559345] Key type .llcrypt registered [ 2154.021341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2155.701135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2157.698223] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:56:05 (1773611765) [ 2157.965300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.975767] alg: No test for adler32 (adler32-zlib) [ 2158.964624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.067771] LNet: 95411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.658409] Key type .llcrypt unregistered [ 2159.662661] Key type ._llcrypt unregistered [ 2168.175537] Key type ._llcrypt registered [ 2168.181200] Key type .llcrypt registered [ 2168.368648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2168.382917] alg: No test for adler32 (adler32-zlib) [ 2169.455475] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2169.461118] LNet: 95667:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2169.474221] LNet: 95667:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2169.528838] LNetError: 95679:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.17@tcp is being used as a gateway but routing feature is not turned on [ 2169.567077] LNetError: 95679:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.17@tcp has gone from up to down [ 2170.196108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.278326] LNet: 95725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.294484] LNetError: 95725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.357019] LNet: Removed LNI 192.168.204.15@tcp [ 2170.447426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.529797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2170.558751] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2170.569467] LNet: 95733:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2170.574990] LNet: 95733:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 2170.581311] LNetError: 95733:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.204.16@tcp is a gateway. Can not delete it [ 2170.591917] LNetError: 95733:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.204.16@tcp is a router. Can not be deleted [ 2170.607558] LNet: 95733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2170.629655] LNetError: 95730:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.20@tcp is being used as a gateway but routing feature is not turned on [ 2170.657111] LNetError: 95730:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2170.672131] LNetError: 95730:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.204.20@tcp has gone from up to down [ 2170.682920] LNetError: 95730:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2179.171795] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:56:26 (1773611786) [ 2182.119427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg415-server [ 2183.342075] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:56:31 (1773611791) [ 2185.496786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg415-server [ 2187.644784] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:56:35 (1773611795) [ 2190.110437] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 2192.081542] Key type lgssc registered [ 2194.034639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.682731] Key type lgssc unregistered [ 2216.974704] LNet: 98155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.981910] LNetError: 98155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.997081] LNet: Removed LNI 192.168.204.15@tcp [ 2217.480426] Key type .llcrypt unregistered [ 2217.482362] Key type ._llcrypt unregistered [ 2231.812236] Key type ._llcrypt registered [ 2231.813968] Key type .llcrypt registered [ 2233.495579] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:57:21 (1773611841) [ 2233.964525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.980190] alg: No test for adler32 (adler32-zlib) [ 2234.912908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.003717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2235.076607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2235.114582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2235.159276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2241.969879] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:57:29 (1773611849) [ 2242.124713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.155155] LNet: 99558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.200325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.313978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.337375] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2243.451524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.489033] LNet: 99706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.502503] LNetError: 99706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.522484] LNet: Removed LNI 192.168.204.15@tcp [ 2243.548241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.636459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2243.684442] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2244.184292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.208212] LNetError: 99734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.222662] LNet: Removed LNI 192.168.204.15@tcp [ 2244.263055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.338135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2244.391457] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2252.118503] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:57:40 (1773611860) [ 2252.607449] LNet: 100325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.614852] LNet: 100325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2252.645751] LNetError: 100325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.672191] LNet: Removed LNI 192.168.204.15@tcp [ 2253.495716] Key type .llcrypt unregistered [ 2253.499765] Key type ._llcrypt unregistered [ 2261.220721] Key type ._llcrypt registered [ 2261.222450] Key type .llcrypt registered [ 2261.599901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2261.628342] alg: No test for adler32 (adler32-zlib) [ 2262.661906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.740918] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2262.743757] LNet: *** cfs_fail_loc=e003, val=100*** [ 2263.738352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 2263.776188] LNet: *** cfs_fail_loc=e003, val=100*** [ 2263.777455] LNet: Skipped 2 previous similar messages [ 2263.781387] Ping reply received with mapping: [ 2263.783092] found metadata reply NI[192.168.204.15@tcp] [ 2263.788791] 0f cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2263.795013] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2263.797560] 5a 5a 5a 5a 5a 5a 5a 5a [ 2263.916798] LNet: 100724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.921700] LNetError: 100724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.936947] LNet: Removed LNI 192.168.204.15@tcp [ 2264.821548] Key type .llcrypt unregistered [ 2264.823924] Key type ._llcrypt unregistered [ 2276.547659] Key type ._llcrypt registered [ 2276.551950] Key type .llcrypt registered [ 2278.119471] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:58:05 (1773611885) [ 2278.814489] Key type .llcrypt unregistered [ 2278.816893] Key type ._llcrypt unregistered [ 2286.398441] Key type ._llcrypt registered [ 2286.403970] Key type .llcrypt registered [ 2286.667766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2286.683294] alg: No test for adler32 (adler32-zlib) [ 2288.001422] LNetError: 101769:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2288.535713] Key type .llcrypt unregistered [ 2288.539871] Key type ._llcrypt unregistered [ 2294.679952] Key type ._llcrypt registered [ 2294.681521] Key type .llcrypt registered [ 2296.309588] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:58:24 (1773611904) [ 2296.954401] Key type .llcrypt unregistered [ 2296.959785] Key type ._llcrypt unregistered [ 2303.825215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2303.827717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2305.407525] Key type ._llcrypt registered [ 2305.416283] Key type .llcrypt registered [ 2305.806509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2305.839301] alg: No test for adler32 (adler32-zlib) [ 2306.935910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.000625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2318.350898] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:58:46 (1773611926) [ 2318.953311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.023981] LNet: 103543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.041608] LNetError: 103543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2319.062946] LNet: Removed LNI 10.1.2.3@tcp [ 2319.106308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.072891] Key type .llcrypt unregistered [ 2320.083720] Key type ._llcrypt unregistered [ 2327.124736] Key type ._llcrypt registered [ 2327.128884] Key type .llcrypt registered [ 2327.510777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2327.541310] alg: No test for adler32 (adler32-zlib) [ 2328.624986] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2328.736248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 2328.928494] LNet: 103832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.935864] LNetError: 103832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.963228] LNet: Removed LNI 192.168.204.15@tcp [ 2330.841586] Key type .llcrypt unregistered [ 2330.848819] Key type ._llcrypt unregistered [ 2344.880356] Key type ._llcrypt registered [ 2344.883184] Key type .llcrypt registered [ 2346.629835] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:59:14 (1773611954) [ 2347.498560] Key type .llcrypt unregistered [ 2347.500063] Key type ._llcrypt unregistered [ 2355.471395] Key type ._llcrypt registered [ 2355.474716] Key type .llcrypt registered [ 2355.753257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2355.782919] alg: No test for adler32 (adler32-zlib) [ 2364.165821] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:59:32 (1773611972) [ 2364.362248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.430110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.521810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.576413] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2365.676134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.709864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2365.737710] LNet: Removed LNI 192.168.204.15@tcp [ 2366.127974] LNet: 105578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.732342] Key type .llcrypt unregistered [ 2366.737478] Key type ._llcrypt unregistered [ 2381.178748] Key type ._llcrypt registered [ 2381.181982] Key type .llcrypt registered [ 2382.959873] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:59:50 (1773611990) [ 2383.557858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.596386] alg: No test for adler32 (adler32-zlib) [ 2384.618700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.740629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2384.789845] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2385.592654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 2385.625578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.15@tcp [ 2385.642304] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2385.645976] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2385.649211] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2385, 0, 0 [ 2385.653811] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2385.655941] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2385.658960] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2385.661902] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2385, 0, 0 [ 2385.666357] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2385.668842] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2385.672347] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2385.675447] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2385, 0, 2435 [ 2385.680198] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2385.682956] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2385.690702] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2385.697897] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2385, 0, 2435 [ 2385.702778] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2393.720305] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:00:01 (1773612001) [ 2394.464609] LNet: 107118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.477683] LNetError: 107118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.506305] LNet: Removed LNI 192.168.204.15@tcp [ 2395.198379] Key type .llcrypt unregistered [ 2395.202211] Key type ._llcrypt unregistered [ 2403.413493] Key type ._llcrypt registered [ 2403.415933] Key type .llcrypt registered [ 2403.766679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2403.818711] alg: No test for adler32 (adler32-zlib) [ 2404.945156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2404.995099] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2414.113302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 2424.598631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.636893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2424.660895] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 2425.853323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2426.032580] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2426.038718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2426.041266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2428.559918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2428.610812] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2442.025655] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:00:50 (1773612050) [ 2442.164442] LNet: 108627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.180208] LNetError: 108627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.185697] LNet: Removed LNI 192.168.204.15@tcp [ 2443.233843] LNet: Removed LNI 192.168.204.15@tcp1 [ 2443.235557] LNetError: 108627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2443.244927] LNetError: 108627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2443.273160] LNet: Removed LNI 10.1.2.3@tcp99 [ 2444.161484] Key type .llcrypt unregistered [ 2444.163724] Key type ._llcrypt unregistered [ 2453.623897] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2453.628071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2456.285341] Key type ._llcrypt registered [ 2456.288432] Key type .llcrypt registered [ 2456.532706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2456.547841] alg: No test for adler32 (adler32-zlib) [ 2457.566093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.673869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2457.741383] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2458.705589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2458.724034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2460.979541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2461.050827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2461.115499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2461.135892] LNetError: 109385:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2461.175461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2461.240959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2461.909826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2462.047934] LNet: 109405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.057871] LNetError: 109405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.074900] LNet: Removed LNI 192.168.204.15@tcp [ 2462.094664] LNet: Removed LNI 10.1.2.3@tcp [ 2462.948597] Key type .llcrypt unregistered [ 2462.950259] Key type ._llcrypt unregistered [ 2477.389437] Key type ._llcrypt registered [ 2477.390759] Key type .llcrypt registered [ 2479.111970] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:01:26 (1773612086) [ 2479.657402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.675421] alg: No test for adler32 (adler32-zlib) [ 2480.634447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.727540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2480.774317] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2482.455946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.15@tcp [ 2482.587623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg415-client@tcp [ 2492.426856] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:01:39 (1773612099) [ 2492.558528] LNet: 110931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.588155] LNetError: 110931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.675231] LNet: Removed LNI 192.168.204.15@tcp [ 2495.078565] Key type .llcrypt unregistered [ 2495.084753] Key type ._llcrypt unregistered [ 2502.731881] Key type ._llcrypt registered [ 2502.736253] Key type .llcrypt registered [ 2502.940527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.968388] alg: No test for adler32 (adler32-zlib) [ 2503.935788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2503.958399] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2503.986447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2511.329095] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:01:59 (1773612119) [ 2513.560427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2516.075589] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:02:03 (1773612123) [ 2518.336748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2520.469725] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:02:08 (1773612128) [ 2520.723095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2520.786526] LNet: 112360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.799947] LNetError: 112360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.827408] LNet: Removed LNI 192.168.204.15@tcp [ 2520.848549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2520.914215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2520.943198] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2521.724592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2521.792382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2529.836437] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:02:17 (1773612137) [ 2530.112847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2530.144654] LNet: 113070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.153730] LNetError: 113070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.181830] LNet: Removed LNI 192.168.204.15@tcp [ 2530.240240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.342504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2530.421316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2531.211365] Key type .llcrypt unregistered [ 2531.216991] Key type ._llcrypt unregistered [ 2544.408987] Key type ._llcrypt registered [ 2544.412197] Key type .llcrypt registered [ 2545.824536] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:02:33 (1773612153) [ 2546.449715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.472556] alg: No test for adler32 (adler32-zlib) [ 2547.385317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2547.440957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2547.463153] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2548.346502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2548.433653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2557.233386] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:02:44 (1773612164) [ 2557.352253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2557.403048] LNet: 114618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.415768] LNetError: 114618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.454400] LNet: Removed LNI 192.168.204.15@tcp [ 2557.492778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2557.580917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2557.642658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2565.633945] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:02:53 (1773612173) [ 2565.847766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2565.896397] LNet: 115207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.933486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.018733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2566.546667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2566.573969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2566.646908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2567.380493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2567.462655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2567.600961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2567.658333] LNet: 115240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2567.692837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2567.796820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2568.559655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2568.612663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2568.680514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2569.282641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2569.342621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2578.120892] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:03:05 (1773612185) [ 2578.449450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2578.495656] LNet: 115841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.531148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2578.645952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2578.677815] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2579.651497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2579.685975] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2580.818472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2580.900926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2581.223838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2581.247657] LNetError: 116120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.308307] LNet: Removed LNI 192.168.204.15@tcp [ 2582.310180] LNetError: 116120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.355304] LNet: Removed LNI 192.168.204.15@tcp2 [ 2582.401386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.518517] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2582.543339] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2582.551434] LNet: 116126:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2583.288558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2583.628562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2583.664480] LNet: 116164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.669465] LNet: 116164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2584.736207] LNet: Removed LNI 192.168.204.15@tcp [ 2584.737873] LNetError: 116164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2584.752659] LNetError: 116164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2584.775619] LNet: Removed LNI 192.168.204.15@tcp2 [ 2584.809977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2584.887862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2584.935710] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2585.800471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2585.837212] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2586.934989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2586.982510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2587.042499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2587.236925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2588.320171] LNet: Removed LNI 192.168.204.15@tcp [ 2589.356898] LNet: Removed LNI 192.168.204.15@tcp2 [ 2589.400170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.489668] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2589.501857] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2589.512170] LNet: 116454:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2590.564982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2590.967058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2590.988631] LNetError: 116492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.993439] LNetError: 116492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2590.998996] LNet: Removed LNI 192.168.204.15@tcp [ 2592.040457] LNet: Removed LNI 192.168.204.15@tcp2 [ 2592.113652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2592.248530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2592.310679] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2593.479884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2593.520288] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2594.650704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2594.756488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2595.010322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2595.027104] LNet: 116771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.030218] LNet: 116771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2596.064555] LNet: Removed LNI 192.168.204.15@tcp [ 2596.082814] LNet: Removed LNI 192.168.204.15@tcp2 [ 2596.115820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2596.210737] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2596.216888] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2596.225194] LNet: 116777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2597.000568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2597.345447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2598.432375] LNet: Removed LNI 192.168.204.15@tcp [ 2598.444437] LNet: Removed LNI 192.168.204.15@tcp2 [ 2598.488709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2598.580395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2598.616357] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2599.306205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2599.323694] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2600.328732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2600.384815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2600.465221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2600.768517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2600.789968] LNetError: 117099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.796168] LNetError: 117099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2600.801723] LNet: Removed LNI 192.168.204.15@tcp [ 2601.831527] LNet: Removed LNI 192.168.204.15@tcp2 [ 2601.913881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2601.976249] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2601.983410] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2601.988783] LNet: 117105:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2602.897606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2603.088690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2603.103307] LNet: Removed LNI 192.168.204.15@tcp [ 2604.176763] LNet: Removed LNI 192.168.204.15@tcp2 [ 2604.236095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2604.349812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2604.423725] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2605.255820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2605.278534] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2606.328877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2606.364933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2606.424587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2606.912907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2607.968258] LNet: Removed LNI 192.168.204.15@tcp [ 2607.987707] LNet: Removed LNI 192.168.204.15@tcp2 [ 2608.067202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.207921] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2608.217089] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2608.223820] LNet: 117445:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2608.924974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2609.323474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2610.400623] LNet: Removed LNI 192.168.204.15@tcp [ 2610.412731] LNet: Removed LNI 192.168.204.15@tcp2 [ 2610.441815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2610.540892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2610.607132] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2611.752415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2611.789466] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2613.080879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2613.122512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2613.149322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2613.204646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2613.599448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2613.622939] LNet: 117799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2613.627812] LNet: 117799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2614.688610] LNet: Removed LNI 192.168.204.15@tcp [ 2615.725229] LNet: Removed LNI 192.168.204.15@tcp2 [ 2615.783948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.875878] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2615.886649] LNet: Added LNI 192.168.204.15@tcp2 [8/256/0/180] [ 2616.628618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2625.409439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2628.282348] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:03:55 (1773612235) [ 2628.433255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2628.487062] LNetError: 118560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.490301] LNetError: 118560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2628.498148] LNet: Removed LNI 192.168.204.15@tcp [ 2629.558425] LNet: Removed LNI 192.168.204.15@tcp2 [ 2629.589440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2629.987805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2630.017448] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2631.195956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2631.220621] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2632.502171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2632.540340] LNet: Removed LNI 192.168.204.15@tcp [ 2632.569148] LNet: Removed LNI 10.1.2.3@tcp [ 2632.610680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2632.729686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2632.779675] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2632.793824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2633.510550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2633.567166] LNet: Removed LNI 192.168.204.15@tcp [ 2633.582217] LNet: Removed LNI 10.1.2.3@tcp [ 2641.364258] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:04:09 (1773612249) [ 2641.536250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2641.594432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2641.737210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2641.942311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2641.960196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2641.965983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2644.642209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2644.705919] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2645.762640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2645.782876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2648.507552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.15@tcp [ 2648.560944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2648.670001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.15@tcp -m PUT -e local_timeout -r 1 [ 2648.773624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2648.881741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2648.892857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2649.268192] LNetError: 119507:0:(lib-move.c:4969:lnet_parse()) 192.168.204.15@tcp, src 192.168.204.15@tcp: Dropping PUT (error -108 looking up sender) [ 2649.952892] LNet: 119966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2649.958761] LNet: 119966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2650.977686] LNet: Removed LNI 192.168.204.15@tcp [ 2650.997412] LNet: Removed LNI 10.1.2.3@tcp [ 2651.619995] Key type .llcrypt unregistered [ 2651.621891] Key type ._llcrypt unregistered [ 2666.918400] Key type ._llcrypt registered [ 2666.923644] Key type .llcrypt registered [ 2668.508960] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:04:36 (1773612276) [ 2669.089465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.109266] alg: No test for adler32 (adler32-zlib) [ 2670.119195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2670.454258] LNet: 120820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2671.154479] Key type .llcrypt unregistered [ 2671.156376] Key type ._llcrypt unregistered [ 2678.001530] Key type ._llcrypt registered [ 2678.003786] Key type .llcrypt registered [ 2678.217859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2678.245364] alg: No test for adler32 (adler32-zlib) [ 2679.320071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2679.372604] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2688.155114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 2688.414921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2689.184826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.115@tcp [ 2689.492452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2689.532566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2689.563582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2689.596480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2689.623114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2689.632341] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 2689.663983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2689.708064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2689.739598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2689.759826] LNet: 121271:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2689.782402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2689.815445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2689.851189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2689.898953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2689.972480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2690.066413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2691.061117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2691.381966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2691.415852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2691.446397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2691.484207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2691.517983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2691.547976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2691.581910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2691.617953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2691.636645] LNet: 121388:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2691.656936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2691.690213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2691.720579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2691.753571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2691.788953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2691.820573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2692.658263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.115@tcp [ 2692.926355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2692.959868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2692.996719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2693.027748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2693.062679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2693.102204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2693.146994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2693.181859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2693.199592] LNet: 121505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2693.216877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2693.254986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2693.281666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2693.329389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2693.380201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2693.447409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2694.194689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2694.496906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2694.568925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2694.627232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2694.657650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2694.731100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2694.824564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2694.896932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2694.973293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2695.014781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2695.057876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2695.100415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2695.163665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2695.221461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2695.261916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2695.756596] LNet: 121669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.764914] LNetError: 121669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.783379] LNet: Removed LNI 192.168.204.15@tcp [ 2696.390913] Key type .llcrypt unregistered [ 2696.399467] Key type ._llcrypt unregistered [ 2717.821312] Key type ._llcrypt registered [ 2717.828045] Key type .llcrypt registered [ 2719.421029] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:05:27 (1773612327) [ 2720.286787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2720.301879] alg: No test for adler32 (adler32-zlib) [ 2721.279342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2721.767431] LNet: 122559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.384314] Key type .llcrypt unregistered [ 2722.386691] Key type ._llcrypt unregistered [ 2729.561667] Key type ._llcrypt registered [ 2729.565629] Key type .llcrypt registered [ 2729.964775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2729.996270] alg: No test for adler32 (adler32-zlib) [ 2730.922525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2730.952986] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2739.710547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.115@tcp [ 2750.449039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2750.505392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2750.538995] LNet: Added LNI 192.168.204.15@tcp1 [8/256/0/180] [ 2751.656173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2752.383531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.115@tcp [ 2752.562826] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 2752.717769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2752.755448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2752.813471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2752.846722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2752.889367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2752.926837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2752.954136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2752.980437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2753.010073] LNet: 123216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2753.027702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2753.066460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2753.101982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2753.153846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2753.205827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2753.265421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2753.980168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2754.247249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2754.275201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2754.306197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2754.342417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2754.366255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2754.391218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2754.419690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2754.443829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2754.461030] LNet: 123333:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2754.474737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2754.498223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2754.526937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2754.553902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2754.584589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2754.608846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2755.178961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.115@tcp [ 2755.387965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2755.406681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2755.436714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2755.463236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2755.489219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2755.513305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2755.542305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2755.574078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2755.589074] LNet: 123451:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2755.607251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2755.631150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2755.660768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2755.686666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2755.715264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2755.744667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2756.362490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2756.647734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2756.673517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2756.698227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2756.724731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2756.743879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2756.761651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2756.789929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2756.812546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2756.843451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2756.866453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2756.889645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2756.916432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2756.936550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2756.966245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2762.177240] LNet: 123649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.185176] LNetError: 123649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.190598] LNet: Removed LNI 192.168.204.15@tcp [ 2762.199899] LNet: Removed LNI 192.168.204.15@tcp1 [ 2762.649635] Key type .llcrypt unregistered [ 2762.653201] Key type ._llcrypt unregistered [ 2781.956285] Key type ._llcrypt registered [ 2781.962067] Key type .llcrypt registered [ 2783.538434] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:06:31 (1773612391) [ 2791.473939] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:06:39 (1773612399) [ 2799.576236] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:06:47 (1773612407) [ 2807.875473] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:06:55 (1773612415) [ 2816.052656] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:07:04 (1773612424) [ 2824.126592] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:07:12 (1773612432) [ 2832.114753] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:07:20 (1773612440) [ 2839.685114] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:07:28 (1773612448) [ 2846.725069] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:07:34 (1773612454) [ 2853.934085] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:07:41 (1773612461) [ 2861.164617] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:07:49 (1773612469) [ 2867.828189] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:07:56 (1773612476) [ 2875.481661] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:08:03 (1773612483) [ 2882.769772] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:08:10 (1773612490) [ 2889.745724] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:08:18 (1773612498) [ 2897.049932] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:08:25 (1773612505) [ 2904.771970] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:08:32 (1773612512) [ 2912.002368] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:08:40 (1773612520) [ 2912.395410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.412235] alg: No test for adler32 (adler32-zlib) [ 2913.328948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2913.487680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2913.519642] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2914.583430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2914.608995] LNet: 134360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.615354] LNetError: 134360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.626795] LNet: Removed LNI 192.168.204.15@tcp [ 2914.649419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2914.726374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2914.752683] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 2920.842802] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:08:49 (1773612529) [ 2927.233894] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:08:55 (1773612535) [ 2933.747394] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:09:01 (1773612541) [ 2941.248153] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:09:09 (1773612549) [ 2947.917796] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:09:16 (1773612556) [ 2955.014610] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:09:23 (1773612563) [ 2962.377914] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:09:30 (1773612570) [ 2968.717882] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:09:37 (1773612577) [ 2974.720994] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:09:43 (1773612583) [ 2981.175950] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:09:49 (1773612589) [ 2987.075927] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:09:55 (1773612595) [ 2993.641596] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:10:01 (1773612601) [ 2999.029255] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2751 sec ========== 18:10:07 (1773612607) [ 2999.390224] LNet: 141824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.394453] LNetError: 141824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.411810] LNet: Removed LNI 192.168.204.15@tcp [ 2999.962743] Key type .llcrypt unregistered [ 2999.965054] Key type ._llcrypt unregistered [ 3010.277457] Key type ._llcrypt registered [ 3010.280604] Key type .llcrypt registered [ 3010.601601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3010.613576] alg: No test for adler32 (adler32-zlib) [ 3011.652468] Lustre: Lustre: Build Version: 2.17.51_1_gce01c51 [ 3011.834299] LNet: Added LNI 192.168.204.15@tcp [8/256/0/180] [ 3013.528666] Key type lgssc registered [ 3014.607938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.758805] Lustre: Mounted lustre-client [ 3073.215772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3085.196461] LustreError: 144281:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9bc6c3664800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3085.260849] Lustre: Unmounted lustre-client [ 3112.036627] Key type lgssc unregistered [ 3112.399917] LNet: 144764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3112.416079] LNetError: 144764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3112.438638] LNet: Removed LNI 192.168.204.15@tcp [ 3113.026364] Key type .llcrypt unregistered [ 3113.033174] Key type ._llcrypt unregistered