[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 435660530 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002328] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.007370] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021672] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026370] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.030502] *** VALIDATE proc *** [ 0.032200] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035092] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042522] debug: unmapping init [mem 0xffffffff9f059000-0xffffffff9f060fff] [ 0.045169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054318] rcu: Hierarchical SRCU implementation. [ 0.056423] smp: Bringing up secondary CPUs ... [ 0.057609] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.063021] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142164] node 0 deferred pages initialised in 74ms [ 0.144102] devtmpfs: initialized [ 0.145286] x86/mm: Memory block size: 128MB [ 0.150391] gcov: version magic: 0x41383552 [ 0.153264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155304] pinctrl core: initialized pinctrl subsystem [ 0.156197] [ 0.156856] ************************************************************* [ 0.157014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158015] ** ** [ 0.159013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161012] ** This means that this kernel is built to expose internal ** [ 0.162014] ** IOMMU data structures, which may compromise security on ** [ 0.163015] ** your system. ** [ 0.164015] ** ** [ 0.165014] ** If you see this message and you are not debugging the ** [ 0.166016] ** kernel, report this immediately to your vendor! ** [ 0.167016] ** ** [ 0.168012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ************************************************************* [ 0.170644] NET: Registered protocol family 16 [ 0.171415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174439] cpuidle: using governor menu [ 0.176779] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180504] PCI: Using configuration type 1 for base access [ 0.182109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194098] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198138] cryptd: max_cpu_qlen set to 1000 [ 0.202045] ACPI: Added _OSI(Module Device) [ 0.203017] ACPI: Added _OSI(Processor Device) [ 0.205024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210987] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215562] ACPI: Interpreter enabled [ 0.217062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219012] ACPI: Using IOAPIC for interrupt routing [ 0.221173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224326] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232449] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243884] acpiphp: Slot [2] registered [ 0.246132] acpiphp: Slot [5] registered [ 0.247253] acpiphp: Slot [6] registered [ 0.249230] acpiphp: Slot [3] registered [ 0.251124] acpiphp: Slot [4] registered [ 0.253204] acpiphp: Slot [7] registered [ 0.255180] acpiphp: Slot [8] registered [ 0.256172] acpiphp: Slot [9] registered [ 0.258119] acpiphp: Slot [10] registered [ 0.260177] acpiphp: Slot [11] registered [ 0.262164] acpiphp: Slot [12] registered [ 0.264165] acpiphp: Slot [13] registered [ 0.266116] acpiphp: Slot [14] registered [ 0.268166] acpiphp: Slot [15] registered [ 0.269150] acpiphp: Slot [16] registered [ 0.271149] acpiphp: Slot [17] registered [ 0.273107] acpiphp: Slot [18] registered [ 0.275101] acpiphp: Slot [19] registered [ 0.276103] acpiphp: Slot [20] registered [ 0.278107] acpiphp: Slot [21] registered [ 0.279097] acpiphp: Slot [22] registered [ 0.281141] acpiphp: Slot [23] registered [ 0.282082] acpiphp: Slot [24] registered [ 0.283063] acpiphp: Slot [25] registered [ 0.285178] acpiphp: Slot [26] registered [ 0.287141] acpiphp: Slot [27] registered [ 0.289123] acpiphp: Slot [28] registered [ 0.291163] acpiphp: Slot [29] registered [ 0.292179] acpiphp: Slot [30] registered [ 0.294205] acpiphp: Slot [31] registered [ 0.296056] PCI host bridge to bus 0000:00 [ 0.298025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312439] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.316613] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320525] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.334054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339056] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348431] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351742] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361840] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371942] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375948] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381741] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411046] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430034] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441440] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470473] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.473365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.476336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.478328] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.479191] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.484647] iommu: Default domain type: Passthrough [ 0.486514] SCSI subsystem initialized [ 0.488139] ACPI: bus type USB registered [ 0.490097] usbcore: registered new interface driver usbfs [ 0.493086] usbcore: registered new interface driver hub [ 0.495078] usbcore: registered new device driver usb [ 0.497188] pps_core: LinuxPPS API ver. 1 registered [ 0.499013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.503059] PTP clock support registered [ 0.505186] EDAC MC: Ver: 3.0.0 [ 0.507143] PCI: Using ACPI for IRQ routing [ 0.508516] NetLabel: Initializing [ 0.510009] NetLabel: domain hash size = 128 [ 0.511015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.513103] NetLabel: unlabeled traffic allowed by default [ 0.515240] vgaarb: loaded [ 0.517236] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.519012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.527909] clocksource: Switched to clocksource kvm-clock [ 0.637817] VFS: Disk quotas dquot_6.6.0 [ 0.639941] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.643159] *** VALIDATE ramfs *** [ 0.644680] *** VALIDATE hugetlbfs *** [ 0.646719] pnp: PnP ACPI init [ 0.649837] pnp: PnP ACPI: found 6 devices [ 0.669128] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.672908] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.675334] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.677556] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.680035] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.682525] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.685337] NET: Registered protocol family 2 [ 0.688047] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.693092] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.696790] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.702274] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.705603] TCP: Hash tables configured (established 65536 bind 65536) [ 0.708749] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.712140] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.715384] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.718858] NET: Registered protocol family 1 [ 0.721717] RPC: Registered named UNIX socket transport module. [ 0.724066] RPC: Registered udp transport module. [ 0.725716] RPC: Registered tcp transport module. [ 0.727399] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.729927] NET: Registered protocol family 44 [ 0.731545] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.733537] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.735435] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737695] PCI: CLS 0 bytes, default 64 [ 0.739341] Unpacking initramfs... [ 2.029393] debug: unmapping init [mem 0xffff8a113cc64000-0xffff8a113ffcffff] [ 2.033086] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.034777] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.036904] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.543662] Initialise system trusted keyrings [ 2.544888] Key type blacklist registered [ 2.546264] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.555080] zbud: loaded [ 2.557376] *** VALIDATE nfs *** [ 2.558727] *** VALIDATE nfs4 *** [ 2.559976] pstore: using deflate compression [ 2.563439] Platform Keyring initialized [ 2.648254] NET: Registered protocol family 38 [ 2.649409] Key type asymmetric registered [ 2.650398] Asymmetric key parser 'x509' registered [ 2.651825] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.654186] io scheduler mq-deadline registered [ 2.656052] io scheduler kyber registered [ 2.657912] io scheduler bfq registered [ 2.659790] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.662872] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.665794] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.668885] ACPI: Power Button [PWRF] [ 2.673253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.677709] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.683878] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.709607] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.735672] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.739294] Non-volatile memory driver v1.3 [ 2.740530] Linux agpgart interface v0.103 [ 2.765607] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.767796] vda: detected capacity change from 0 to 68911104 [ 2.779735] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.781664] vdb: detected capacity change from 0 to 1073741824 [ 2.785293] libphy: Fixed MDIO Bus: probed [ 2.792271] usbcore: registered new interface driver usbserial_generic [ 2.794041] usbserial: USB Serial support registered for generic [ 2.795762] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.799376] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.800896] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.802688] mousedev: PS/2 mouse device common for all mice [ 2.804730] rtc_cmos 00:05: RTC can wake from S4 [ 2.807473] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.810390] rtc_cmos 00:05: registered as rtc0 [ 2.812973] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.813026] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.814488] intel_pstate: CPU model not supported [ 2.818793] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.822634] hid: raw HID events driver (C) Jiri Kosina [ 2.824647] usbcore: registered new interface driver usbhid [ 2.826523] usbhid: USB HID core driver [ 2.828100] drop_monitor: Initializing network drop monitor service [ 2.830142] Initializing XFRM netlink socket [ 2.832806] NET: Registered protocol family 10 [ 2.835500] Segment Routing with IPv6 [ 2.836716] NET: Registered protocol family 17 [ 2.837975] mpls_gso: MPLS GSO support [ 2.842260] RAS: Correctable Errors collector initialized. [ 2.843804] AVX version of gcm_enc/dec engaged. [ 2.844908] AES CTR mode by8 optimization enabled [ 2.905948] sched_clock: Marking stable (2905917568, 0)->(3832148432, -926230864) [ 2.908200] registered taskstats version 1 [ 2.909914] Loading compiled-in X.509 certificates [ 2.911870] zswap: loaded using pool lzo/zbud [ 2.932363] Key type big_key registered [ 2.941947] Key type encrypted registered [ 2.943988] ima: No TPM chip found, activating TPM-bypass! [ 2.945823] ima: Allocated hash algorithm: sha1 [ 2.947161] ima: No architecture policies found [ 2.948396] evm: Initialising EVM extended attributes: [ 2.950036] evm: security.selinux [ 2.951051] evm: security.ima [ 2.952142] evm: security.capability [ 2.953969] evm: HMAC attrs: 0x1 [ 2.956326] rtc_cmos 00:05: setting system clock to 2026-03-15 20:33:21 UTC (1773606801) [ 2.963106] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 2.966589] debug: unmapping init [mem 0xffffffff9ed82000-0xffffffff9f058fff] [ 2.975090] Write protecting the kernel read-only data: 28672k [ 2.978804] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 2.982155] debug: unmapping init [mem 0xffffffff9dd14000-0xffffffff9ddfffff] [ 3.011504] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.021429] systemd[1]: Detected virtualization kvm. [ 3.022808] systemd[1]: Detected architecture x86-64. [ 3.023997] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.049284] systemd[1]: No hostname configured. [ 3.050463] systemd[1]: Set hostname to . [ 3.051806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.053534] systemd[1]: Initializing machine ID from random generator. [ 3.173315] random: systemd: uninitialized urandom read (16 bytes read) [ 3.175171] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.178403] random: systemd: uninitialized urandom read (16 bytes read) [ 3.180174] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.184301] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.752340] device-mapper: uevent: version 1.0.3 [ 3.754804] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.365181] virtio_net virtio0 ens2: renamed from eth0 [ 4.427295] scsi host0: ata_piix [ 4.451376] scsi host1: ata_piix [ 4.453059] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.455873] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.630626] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.459219] random: crng init done [ 9.460718] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.720986] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.879602] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.134649] SELinux: Disabled at runtime. [ 11.190893] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.196484] systemd[1]: Detected virtualization kvm. [ 11.198031] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.603239] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.606076] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.610707] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.614640] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.618368] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.630911] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.636182] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ 11.727957] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.067160] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.374735] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.397250] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.526686] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.555109] EDAC sbridge: Ver: 1.1.2 [ 13.558533] Key type dns_resolver registered [ 13.887323] NFS: Registering the id_resolver key type [ 13.889035] Key type id_resolver registered [ 13.890811] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg332-client login: [ 42.954886] libcfs: loading out-of-tree module taints kernel. [ 43.198721] Key type ._llcrypt registered [ 43.200721] Key type .llcrypt registered [ 43.451565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.457538] alg: No test for adler32 (adler32-zlib) [ 44.537302] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 44.913578] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 46.543178] Key type lgssc registered [ 47.337381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.791612] hrtimer: interrupt took 8178567 ns [ 170.848680] Lustre: Mounted lustre-client [ 176.139046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.923378] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing check_logdir /tmp/testlogs/ [ 196.586448] Lustre: lustre-OST0000-osc-ffff8a1191d32800: disconnect after 24s idle [ 200.989243] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing yml_node [ 206.426389] Lustre: DEBUG MARKER: Client: 2.17.51.2 [ 208.826934] Lustre: DEBUG MARKER: MDS: 2.17.51.2 [ 211.831981] Lustre: DEBUG MARKER: OSS: 2.17.51.2 [ 213.919193] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Mar 15 16:36:50 EDT 2026 [ 234.479794] LustreError: 7902:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8a1191d32800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 234.535934] Lustre: Unmounted lustre-client [ 265.776550] Key type lgssc unregistered [ 266.140387] LNet: 8351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.148518] LNetError: 8351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 266.167050] LNet: Removed LNI 192.168.203.32@tcp [ 267.170186] Key type .llcrypt unregistered [ 267.172896] Key type ._llcrypt unregistered [ 301.078583] Key type ._llcrypt registered [ 301.082175] Key type .llcrypt registered [ 301.340318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 301.359429] alg: No test for adler32 (adler32-zlib) [ 302.218531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 302.289658] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 302.338188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 307.372506] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 307.725630] LNet: 9229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.739913] LNetError: 9229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.798563] LNet: Removed LNI 192.168.203.32@tcp [ 308.529552] Key type .llcrypt unregistered [ 308.536412] Key type ._llcrypt unregistered [ 318.591568] Key type ._llcrypt registered [ 318.592902] Key type .llcrypt registered [ 320.033207] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:38:36 (1773607116) [ 320.652117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 320.669598] alg: No test for adler32 (adler32-zlib) [ 321.593065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.636867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.650513] LNet: 9643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.674546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.715195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 321.727912] LNet: 9650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 330.477650] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:38:47 (1773607127) [ 330.773245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.831551] LNet: 10228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.877544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.013817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 339.337586] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:38:56 (1773607136) [ 339.534591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.605343] LNet: 10809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.696476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.901661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 348.543715] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:39:05 (1773607145) [ 348.858524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.895641] LNet: 11399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.927590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.028876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 357.253817] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:39:14 (1773607154) [ 357.546847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.582640] LNet: 11985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.612966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.727545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 358.356982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 366.541132] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:39:23 (1773607163) [ 366.855811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.919551] LNet: 12578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.971713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.086918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 367.624572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 368.193732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 375.884687] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:39:32 (1773607172) [ 376.342725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.406771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.513669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 384.609748] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:39:41 (1773607181) [ 384.741557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.763940] LNet: 13754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.773265] LNet: 13754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 384.811310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.891750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 384.922891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.463798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 385.523225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 386.093457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 386.152901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 386.673193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 386.704428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 387.285365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 387.323123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 387.910494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 387.958765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 388.499382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 388.543370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.080347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 389.114638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 389.737635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 389.802080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 390.304941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 390.339948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 390.898867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 390.926409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 391.415840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 391.500243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 392.179067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 392.231442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 400.565912] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:39:57 (1773607197) [ 400.908595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.989901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.119858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 401.185865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 408.802962] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:40:05 (1773607205) [ 408.968659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.995289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.038796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.060912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 415.695921] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:40:12 (1773607212) [ 415.931760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.970389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.020993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 416.078322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 424.797709] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:40:21 (1773607221) [ 424.962074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.036039] LNet: 16196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.040277] LNet: 16196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 425.093165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.272717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.373835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 433.680984] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:40:30 (1773607230) [ 433.793841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.862761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.000207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 434.057984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 441.753827] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:40:38 (1773607238) [ 441.917466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.027956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.185801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 442.219087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 449.365732] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:40:46 (1773607246) [ 449.828044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.883981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.032275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.475938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.577150] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:40:54 (1773607254) [ 458.090195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.158928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.516697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.768542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.644170] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:41:02 (1773607262) [ 465.941952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.977158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.100469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.382683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.444163] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:41:11 (1773607271) [ 474.568050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.635846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.344763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.768938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.575885] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:41:22 (1773607282) [ 486.765562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.854824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.095927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.209678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.540680] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:41:33 (1773607293) [ 496.772333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.816529] LNet: 21036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.827798] LNet: 21036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 496.899683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.997280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.075179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.610389] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:41:42 (1773607302) [ 506.025980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.064118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.138814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 513.874903] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:41:51 (1773607311) [ 514.227480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.296246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.409719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 523.298549] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:42:00 (1773607320) [ 523.458835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.514758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.572619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 524.189582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 532.306901] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:42:09 (1773607329) [ 532.870627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.965118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.176613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 533.261722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 542.029991] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:42:18 (1773607338) [ 542.221303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.276027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.335400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 542.370070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 549.502791] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:42:26 (1773607346) [ 549.613532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.706319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.787872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 549.833249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 557.229093] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:42:34 (1773607354) [ 557.363895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.421246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.532176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 557.573837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.660426] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:42:41 (1773607361) [ 564.795406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.822834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.857684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 564.923943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 564.941840] LNetError: 25764:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 564.979609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 565.015837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 572.827883] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:42:50 (1773607370) [ 573.236765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.291320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.405333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KaRQq3uOlc [ 573.526320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KaRQq3uOlc [ 573.603848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KaRQq3uOlc [ 582.319185] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:42:58 (1773607378) [ 582.420742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.570338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.649286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 582.680287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 582.747179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 591.294372] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:43:08 (1773607388) [ 591.525994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.612671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.730848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 594.648030] LNet: Message forwarding will be enabled in 2s [ 597.727177] LNet: Message forwarding enabled [ 598.363829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.408096] LNet: Message forwarding will stop in 2s to 6s [ 602.109387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 603.831114] LNet: Message forwarding disabled [ 604.952815] LNet: Message forwarding will be enabled in 2s [ 608.995960] LNet: Message forwarding enabled [ 612.280295] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:43:29 (1773607409) [ 612.947347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.336696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.483753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 624.336398] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:43:40 (1773607420) [ 624.529382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.636493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.830987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 624.935340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 624.983182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 625.039678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 625.108974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 625.181693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 625.258511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 625.348777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 625.439626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 625.480952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 625.513773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 625.556806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 625.605559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 625.636391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 625.674325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 625.711864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 625.742296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 625.776966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 625.827921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 625.874939] LNetError: 28829:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 635.137882] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:43:52 (1773607432) [ 635.493705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.511981] LNet: 29401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.517397] LNet: 29401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 635.544672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.640380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.122826] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:44:01 (1773607441) [ 646.814685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg332-server [ 649.505533] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:44:06 (1773607446) [ 649.644326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.696531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.788648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.808677] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 652.343749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[33-37]@tcp [ 652.388275] LNet: 30547:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 652.453175] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 652.505919] LNetError: 30336:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.34@tcp has gone from up to down [ 653.477467] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.37@tcp is being used as a gateway but routing feature is not turned on [ 653.498562] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 654.500311] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 654.518358] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 656.558685] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 656.590039] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 660.733595] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 660.747032] LNetError: 30336:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 661.870466] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:44:18 (1773607458) [ 663.907503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg332-server [ 665.897286] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:44:22 (1773607462) [ 665.975317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.013637] LNetError: 31475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.068950] LNet: Removed LNI 192.168.203.32@tcp [ 666.134255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.233415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.270636] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 667.036827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 667.094037] LNet: 31600:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 667.099610] LNet: 31600:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 667.125801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 667.129069] LNetError: 31486:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.37@tcp has gone from up to down [ 667.158664] LNetError: 31486:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 674.688176] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:44:31 (1773607471) [ 674.904135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.939093] LNetError: 32190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.964383] LNet: Removed LNI 192.168.203.32@tcp [ 675.003583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.103733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 675.144509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.167497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.231277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.299122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.343126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.409640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 675.520955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.562576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.653619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.778847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.817499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.906898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 676.033821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.098667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.175778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.305530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.353518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.410261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 676.506532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.534824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.616876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.701243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.738486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.795639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 676.882514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.952263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.030948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.134390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.182087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.244279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 677.338641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.382121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.514091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.380577] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:44:41 (1773607481) [ 686.294342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg332-server [ 687.885395] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:44:45 (1773607485) [ 689.827155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg332-server [ 691.344242] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:44:48 (1773607488) [ 692.195595] Key type .llcrypt unregistered [ 692.198082] Key type ._llcrypt unregistered [ 698.025462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 698.306362] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 698.313935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 698.323045] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 700.973832] Key type ._llcrypt registered [ 700.980294] Key type .llcrypt registered [ 701.227962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.251633] alg: No test for adler32 (adler32-zlib) [ 702.129691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.176087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.203939] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 702.735875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 702.757934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.462661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 704.479346] LNet: Removed LNI 10.1.2.3@tcp [ 704.556677] LNet: 34318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.559920] LNetError: 34318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.569163] LNet: Removed LNI 192.168.203.32@tcp [ 705.154562] Key type .llcrypt unregistered [ 705.157366] Key type ._llcrypt unregistered [ 716.428111] Key type ._llcrypt registered [ 716.430224] Key type .llcrypt registered [ 717.868889] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:45:14 (1773607514) [ 718.274285] Key type .llcrypt unregistered [ 718.277151] Key type ._llcrypt unregistered [ 725.108068] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.407757] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.410167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.516024] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 728.324120] Key type ._llcrypt registered [ 728.328429] Key type .llcrypt registered [ 728.650145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.681969] alg: No test for adler32 (adler32-zlib) [ 729.631219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.688932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 729.725968] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 730.549472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.564283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.990762] LNet: 35812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.997836] LNetError: 35812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.007619] LNet: Removed LNI 192.168.203.32@tcp [ 733.020116] LNet: Removed LNI 10.1.2.3@tcp [ 733.622680] Key type .llcrypt unregistered [ 733.624563] Key type ._llcrypt unregistered [ 746.313755] Key type ._llcrypt registered [ 746.315412] Key type .llcrypt registered [ 747.980945] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:45:44 (1773607544) [ 748.705601] Key type .llcrypt unregistered [ 748.711179] Key type ._llcrypt unregistered [ 756.288746] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.424243] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.435860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.446686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.161461] Key type ._llcrypt registered [ 759.166241] Key type .llcrypt registered [ 759.373602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.392898] alg: No test for adler32 (adler32-zlib) [ 760.340235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.413555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 760.451415] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.323842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.348389] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.649420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 764.673928] LNet: Removed LNI 10.1.2.3@tcp [ 764.691666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 764.728886] LNet: Removed LNI 10.1.2.31@tcp [ 764.748547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.769981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.584515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 765.607615] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 766.969394] LNet: 37732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.975447] LNetError: 37732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 766.984362] LNet: Removed LNI 10.1.2.3@tcp [ 766.993635] LNet: Removed LNI 10.1.2.31@tcp [ 767.839377] Key type .llcrypt unregistered [ 767.841163] Key type ._llcrypt unregistered [ 781.900625] Key type ._llcrypt registered [ 781.908270] Key type .llcrypt registered [ 783.704851] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:46:20 (1773607580) [ 784.278513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.290423] alg: No test for adler32 (adler32-zlib) [ 785.282754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.331517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.350602] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 786.414448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.445481] LNet: 38685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.452431] LNetError: 38685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.463301] LNet: Removed LNI 192.168.203.32@tcp [ 786.498353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.547848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.32@tcp [ 786.576882] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 787.393589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.32@tcp [ 787.431782] LNet: Removed LNI 192.168.203.32@tcp [ 787.497400] LNet: 38820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.315492] Key type .llcrypt unregistered [ 788.317145] Key type ._llcrypt unregistered [ 801.355981] Key type ._llcrypt registered [ 801.357864] Key type .llcrypt registered [ 802.526470] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:46:39 (1773607599) [ 804.472956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg332-server [ 805.958640] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:46:43 (1773607603) [ 806.508961] Key type .llcrypt unregistered [ 806.515196] Key type ._llcrypt unregistered [ 812.743602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.884689] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.895090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.919365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.475476] Key type ._llcrypt registered [ 815.477650] Key type .llcrypt registered [ 815.634813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.648420] alg: No test for adler32 (adler32-zlib) [ 816.505425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.547452] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 817.099237] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 819.693806] LNet: 40658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.701388] LNetError: 40658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.708698] LNet: Removed LNI 192.168.203.32@tcp1 [ 819.720797] LNet: Removed LNI 10.1.2.3@tcp2 [ 820.561579] Key type .llcrypt unregistered [ 820.570595] Key type ._llcrypt unregistered [ 834.406289] Key type ._llcrypt registered [ 834.408940] Key type .llcrypt registered [ 836.332929] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:47:12 (1773607632) [ 836.744301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.763264] alg: No test for adler32 (adler32-zlib) [ 837.854647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.909531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.972380] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 838.723552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.736532] LNet: 41627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.741663] LNetError: 41627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.751852] LNet: Removed LNI 192.168.203.32@tcp [ 838.772836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.865939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 838.891053] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 845.791726] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:47:23 (1773607643) [ 846.241694] LNet: 42218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.259793] LNetError: 42218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.279488] LNet: Removed LNI 192.168.203.32@tcp [ 847.273387] Key type .llcrypt unregistered [ 847.275481] Key type ._llcrypt unregistered [ 854.914189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.125796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.146544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.149129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.925765] Key type ._llcrypt registered [ 857.927879] Key type .llcrypt registered [ 858.190643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.237492] alg: No test for adler32 (adler32-zlib) [ 859.172258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.242797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.264719] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 859.968213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 859.982378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.235070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.269966] LNet: 42959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.276832] LNetError: 42959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.282896] LNet: Removed LNI 192.168.203.32@tcp [ 862.296258] LNet: Removed LNI 10.1.2.3@tcp [ 862.329529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.433046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 862.478114] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 862.485246] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.074603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.093862] LNet: 42986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.098664] LNetError: 42986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.102358] LNetError: 42986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 863.116658] LNet: Removed LNI 192.168.203.32@tcp [ 863.129832] LNet: Removed LNI 10.1.2.3@tcp [ 863.168327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.279260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 863.336761] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 863.353814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.004054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 864.033139] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 865.019398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.031456] LNet: 43148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.036189] LNetError: 43148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.041166] LNetError: 43148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 865.048313] LNet: Removed LNI 192.168.203.32@tcp [ 865.051283] LNet: Removed LNI 10.1.2.3@tcp [ 865.064794] LNet: Removed LNI 192.168.203.32@tcp2 [ 865.087351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.141626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 865.162477] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 865.165656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.169822] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 872.048742] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:47:49 (1773607669) [ 872.167670] LNet: 43737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.182950] LNetError: 43737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.192315] LNetError: 43737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 872.211708] LNet: Removed LNI 192.168.203.32@tcp [ 872.217782] LNet: Removed LNI 10.1.2.3@tcp [ 872.246269] LNet: Removed LNI 192.168.203.32@tcp2 [ 872.811761] Key type .llcrypt unregistered [ 872.814136] Key type ._llcrypt unregistered [ 878.948590] Key type ._llcrypt registered [ 878.950494] Key type .llcrypt registered [ 879.101906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.115867] alg: No test for adler32 (adler32-zlib) [ 880.001409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.045903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.086993] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 880.911803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.925292] LNet: 44141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.935376] LNetError: 44141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.956324] LNet: Removed LNI 192.168.203.32@tcp [ 880.974040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.025935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 881.050555] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 881.564619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.587802] LNet: 44171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.594746] LNetError: 44171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.610737] LNet: Removed LNI 192.168.203.32@tcp [ 881.630818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.684417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 881.711540] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 882.392736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.428663] LNet: Removed LNI 192.168.203.32@tcp [ 882.488559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.551499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 882.577786] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 882.877961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.027312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 883.039165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 883.058544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.575924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.608239] LNet: 44278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.618961] LNet: 44278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 885.627735] LNetError: 44278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.646051] LNetError: 44278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.678674] LNet: Removed LNI 192.168.203.32@tcp [ 885.729979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.824225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 885.865632] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 892.762209] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:48:10 (1773607690) [ 893.085503] LNet: 44876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.093900] LNetError: 44876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.113314] LNet: Removed LNI 192.168.203.32@tcp [ 893.698835] Key type .llcrypt unregistered [ 893.701397] Key type ._llcrypt unregistered [ 899.482989] Key type ._llcrypt registered [ 899.488499] Key type .llcrypt registered [ 899.664983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.678555] alg: No test for adler32 (adler32-zlib) [ 900.629929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.712860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.745692] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 901.650512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.662906] LNet: 45275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.666815] LNetError: 45275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.679382] LNet: Removed LNI 192.168.203.32@tcp [ 901.700418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.769527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 901.814625] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 908.602524] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:48:25 (1773607705) [ 908.707147] LNet: 45861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.727895] LNetError: 45861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.758957] LNet: Removed LNI 192.168.203.32@tcp [ 909.524637] Key type .llcrypt unregistered [ 909.526202] Key type ._llcrypt unregistered [ 915.645769] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 915.959182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 915.961958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 915.964339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.670727] Key type ._llcrypt registered [ 918.672150] Key type .llcrypt registered [ 918.915381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.929936] alg: No test for adler32 (adler32-zlib) [ 919.982384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.090371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 920.114132] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 921.421768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.461162] LNet: 46319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.476795] LNetError: 46319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.496503] LNet: Removed LNI 192.168.203.32@tcp [ 921.522568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.581699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 921.615106] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 930.880792] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:48:47 (1773607727) [ 931.376733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.401431] LNet: 46918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.409260] LNetError: 46918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.421457] LNet: Removed LNI 192.168.203.32@tcp [ 931.453066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.591902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 938.736375] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:48:55 (1773607735) [ 939.033448] LNet: 47496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.006736] Key type .llcrypt unregistered [ 940.010836] Key type ._llcrypt unregistered [ 948.447031] Key type ._llcrypt registered [ 948.453753] Key type .llcrypt registered [ 948.738881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.762851] alg: No test for adler32 (adler32-zlib) [ 949.793372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.829667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.861314] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 951.062762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.088876] LNet: 47915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.101754] LNetError: 47915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.127058] LNet: Removed LNI 192.168.203.32@tcp [ 951.159880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.372653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 951.400806] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 958.914231] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:49:16 (1773607756) [ 959.211374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.233667] LNet: 48508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.244366] LNetError: 48508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.270675] LNet: Removed LNI 192.168.203.32@tcp [ 959.304492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.413769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 966.937544] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:49:24 (1773607764) [ 968.776697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 970.623596] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:49:27 (1773607767) [ 972.200601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 973.958946] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:49:31 (1773607771) [ 976.119980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 978.152503] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:49:35 (1773607775) [ 979.708394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 981.666522] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:49:38 (1773607778) [ 983.899216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 986.007385] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:49:42 (1773607782) [ 986.307624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.344782] LNet: 50564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.374700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.452082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.492187] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 987.548234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.32 [ 987.587152] LNet: 50697:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 987.599136] LNet: Removed LNI 192.168.203.32@tcp [ 993.788824] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:49:50 (1773607790) [ 994.075954] LNet: 51268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.087471] LNetError: 51268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.114684] LNet: Removed LNI 192.168.203.32@tcp [ 994.754897] Key type .llcrypt unregistered [ 994.757455] Key type ._llcrypt unregistered [ 1001.393816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1001.601765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1001.610369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1001.614322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1004.361791] Key type ._llcrypt registered [ 1004.371470] Key type .llcrypt registered [ 1004.610190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.633791] alg: No test for adler32 (adler32-zlib) [ 1006.186715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1012.777676] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:50:10 (1773607810) [ 1013.327354] LNet: 52173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.333553] LNetError: 52173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.364986] LNet: Removed LNI 10.1.2.3@tcp [ 1014.042428] Key type .llcrypt unregistered [ 1014.044380] Key type ._llcrypt unregistered [ 1022.051115] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1022.287402] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1022.291337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1022.298169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1024.933913] Key type ._llcrypt registered [ 1024.935666] Key type .llcrypt registered [ 1025.096764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.116866] alg: No test for adler32 (adler32-zlib) [ 1026.069737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1033.077527] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:50:30 (1773607830) [ 1033.509480] LNet: 53066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.515377] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1033.543933] LNet: Removed LNI 10.1.2.3@tcp [ 1034.046739] Key type .llcrypt unregistered [ 1034.050542] Key type ._llcrypt unregistered [ 1042.334371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1042.678597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1042.685650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1042.691960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.360909] Key type ._llcrypt registered [ 1045.367395] Key type .llcrypt registered [ 1045.547965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.568763] alg: No test for adler32 (adler32-zlib) [ 1046.610378] LNet: 53367:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1046.629064] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1053.778393] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:50:50 (1773607850) [ 1054.017517] LNet: 53959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.025271] LNetError: 53959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.061118] LNet: Removed LNI 10.1.2.3@tcp [ 1054.721521] Key type .llcrypt unregistered [ 1054.725066] Key type ._llcrypt unregistered [ 1061.896481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1062.229661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1062.234709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1062.240658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1065.280061] Key type ._llcrypt registered [ 1065.282258] Key type .llcrypt registered [ 1065.595084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.617620] alg: No test for adler32 (adler32-zlib) [ 1066.581869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.715970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.197763] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:51:13 (1773607873) [ 1077.271916] LNet: 55189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.276833] LNetError: 55189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1077.297274] LNet: Removed LNI 10.1.2.3@tcp [ 1078.096943] Key type .llcrypt unregistered [ 1078.100359] Key type ._llcrypt unregistered [ 1085.305593] Key type ._llcrypt registered [ 1085.311559] Key type .llcrypt registered [ 1085.556490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.576499] alg: No test for adler32 (adler32-zlib) [ 1086.559843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1086.595519] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1095.825209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1096.562974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 1096.606634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 1096.679985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1096.711630] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1096.863657] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1097.056933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.081624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1097.110133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.146908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.205183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.238465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.286301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.337947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.369570] LNet: 55626:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.385718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.446728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.481433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.516472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.553578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.588636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.618336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.119709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 1098.151183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 1098.181325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1098.201460] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1098.452154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.477593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1098.538613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.581634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.616480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.661961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.724620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.803692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.833499] LNet: 55735:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.860703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.900624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.957726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.009670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.068641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.146297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.196597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1099.842976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 1099.896652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 1099.931101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1099.950799] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1100.198625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.238775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.287865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.323362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.361882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.411292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.470676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.543963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.580923] LNet: 55844:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.610333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.657327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.709916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.762635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.819141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.880449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.916317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1101.626525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 1101.682993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 1101.746345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1102.033803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.074674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.100695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.143408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.165961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.227518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.308813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.376800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.406970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.459390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.526974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.579321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.624781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.683215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.718243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.087871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 1103.121293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 1103.150614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1103.175884] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1103.184359] LNet: Skipped 1 previous similar message [ 1103.445741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.491608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1103.574506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.625233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.680582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.745461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.815787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.886381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.909990] LNet: 56063:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.916279] LNet: 56063:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1103.937810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.971915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.991211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.006524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.023236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.038208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.053103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1104.444040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 1104.470419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 1104.497796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1104.705453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.719584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.744950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.768784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.795667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.816240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.835954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.855089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.876979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.895924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.909653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.928626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.952918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.968791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.981693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.175331] LNet: 56213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.180295] LNetError: 56213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.196341] LNet: Removed LNI 192.168.203.32@tcp [ 1105.770614] Key type .llcrypt unregistered [ 1105.774098] Key type ._llcrypt unregistered [ 1128.652385] Key type ._llcrypt registered [ 1128.654939] Key type .llcrypt registered [ 1130.167531] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:52:07 (1773607927) [ 1131.312820] Key type .llcrypt unregistered [ 1131.314549] Key type ._llcrypt unregistered [ 1138.109562] Key type ._llcrypt registered [ 1138.111245] Key type .llcrypt registered [ 1138.380736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.387254] alg: No test for adler32 (adler32-zlib) [ 1139.317532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.367565] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1147.570073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1158.652727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1158.722303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1158.763481] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1160.482423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 1160.512927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 1160.547275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 1160.581288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 1160.611603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 1160.660056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 1160.704986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 1160.739065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 1160.814938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1160.843827] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1160.852256] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1161.085642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.110900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.132398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.164118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.195412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.221627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.248799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.292268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.332280] LNet: 57680:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.370950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.425905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.476301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.504148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.526349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.560237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.589601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.103903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 1162.136938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 1162.182930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 1162.244734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 1162.321084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 1162.390233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 1162.425625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 1162.470264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 1162.512833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1162.539802] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1162.546391] LNet: Skipped 2 previous similar messages [ 1162.853898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.902589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1162.949630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.983055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.017759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.052834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.091518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.150147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.171973] LNet: 57801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.208581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.255831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.306978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.338070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.381620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.425451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.466368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.112812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 1164.164923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 1164.243370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 1164.294958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 1164.372527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 1164.433787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 1164.488462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 1164.563760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 1164.627910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1164.645257] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1164.649458] LNet: Skipped 2 previous similar messages [ 1164.901811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.957610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1164.997246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.025341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.059797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.090648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.129241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.158495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.170577] LNet: 57923:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.199869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.225846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.260235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.289872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.325230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.368710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.398180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.949670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 1165.998550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 1166.047424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 1166.095460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 1166.169672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 1166.227493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 1166.274375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 1166.318785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 1166.366059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1166.637549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.661942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1166.695392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.728612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.751860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.785886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.818467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.858553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.888408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.922139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.947805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.977160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.018652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.066744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.110995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.575697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 1167.604881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 1167.631888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 1167.661497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 1167.695876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 1167.724263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 1167.750215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 1167.777676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 1167.805669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1167.824950] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1167.830339] LNet: Skipped 5 previous similar messages [ 1168.013830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.039284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1168.059761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.087352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.108644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.132222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.157621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.191550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.207529] LNet: 58165:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.212127] LNet: 58165:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1168.235851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.268378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.299401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.340031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.401317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.435648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.476969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.202366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 1169.241679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 1169.279660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 1169.319477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 1169.347895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 1169.383831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 1169.427380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 1169.481492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 1169.549932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1169.817829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.855487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1169.907336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.962777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.008218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.048684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.100656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.183039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.243216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.326922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.347827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.390661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.430664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.466316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.491543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.569458] LNet: 58362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.580516] LNetError: 58362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.593121] LNet: Removed LNI 192.168.203.32@tcp [ 1175.607915] LNet: Removed LNI 192.168.203.32@tcp1 [ 1176.088550] Key type .llcrypt unregistered [ 1176.090097] Key type ._llcrypt unregistered [ 1193.727750] Key type ._llcrypt registered [ 1193.736334] Key type .llcrypt registered [ 1195.611895] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:53:12 (1773607992) [ 1196.478450] Key type .llcrypt unregistered [ 1196.486354] Key type ._llcrypt unregistered [ 1202.373349] Key type ._llcrypt registered [ 1202.377606] Key type .llcrypt registered [ 1202.586122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.609338] alg: No test for adler32 (adler32-zlib) [ 1203.605871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.625587] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1210.923130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1211.621770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1211.657709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1211.680177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1211.694084] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1211.927711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.950735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1211.973639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.003929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.023848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.040735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.059812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.085238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.114369] LNet: 59611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.140690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.167558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.191179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.215624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.237679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.260689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.290854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1212.870634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1212.903992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1212.955852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1212.988454] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1213.238894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.276968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.320729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.353869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.402582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.427244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.468958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.525809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.546351] LNet: 59720:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.593771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.637886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.662422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.688836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.717401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.737656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.758790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.194193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1214.215894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1214.236436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1214.251281] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1214.465632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.519822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.578108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.623445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.656866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.684421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.723153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.753581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.782455] LNet: 59829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.819706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.848813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.892301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.930281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.956505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.002343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.048127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.323932] LNet: 59870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.330901] LNetError: 59870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.352217] LNet: Removed LNI 192.168.203.32@tcp [ 1215.751572] Key type .llcrypt unregistered [ 1215.753913] Key type ._llcrypt unregistered [ 1231.189438] Key type ._llcrypt registered [ 1231.191969] Key type .llcrypt registered [ 1232.339531] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:53:49 (1773608029) [ 1232.941484] Key type .llcrypt unregistered [ 1232.943361] Key type ._llcrypt unregistered [ 1239.332356] Key type ._llcrypt registered [ 1239.334480] Key type .llcrypt registered [ 1239.546531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.557258] alg: No test for adler32 (adler32-zlib) [ 1240.557751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.588353] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1247.857768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1257.712386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.751820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1257.791524] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1259.395924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1259.433466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 1259.486526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1259.551935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 1259.596615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1259.631480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 1259.687841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1259.740586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 1259.780962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1259.808550] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1259.818739] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1260.024928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.085368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1260.135697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.180089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.216829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.242955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.273019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.324164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.344990] LNet: 61337:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.379130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.406777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.432959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.454403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.476537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.502148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.520980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.548921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.028473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1261.057550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 1261.094276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1261.124835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 1261.167839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1261.212513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 1261.243569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1261.280749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 1261.331515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1261.355156] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1261.363641] LNet: Skipped 2 previous similar messages [ 1261.613641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.637530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1261.656859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.692803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.722291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.750811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.783787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.819669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.842903] LNet: 61460:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.863309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.901272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.946542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.990561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.028984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.060982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.091728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1262.114935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.538313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1262.560374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 1262.600973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1262.629917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 1262.670350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1262.722165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 1262.780397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1262.825128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 1262.874110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1262.913502] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1263.172671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.201366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1263.223749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.251291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.302563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.330536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.359757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.387722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.403409] LNet: 61583:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.424562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.452651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.481671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.517172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.551840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.592433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.649711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1263.682437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.139396] LNet: 61661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.148360] LNetError: 61661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.155114] LNet: Removed LNI 192.168.203.32@tcp [ 1267.177282] LNet: Removed LNI 192.168.203.32@tcp1 [ 1267.617717] Key type .llcrypt unregistered [ 1267.623722] Key type ._llcrypt unregistered [ 1282.980195] Key type ._llcrypt registered [ 1282.981559] Key type .llcrypt registered [ 1284.074453] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:54:41 (1773608081) [ 1284.494200] Key type .llcrypt unregistered [ 1284.505070] Key type ._llcrypt unregistered [ 1290.578350] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1290.730959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1290.733562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1290.743736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1293.486856] Key type ._llcrypt registered [ 1293.489478] Key type .llcrypt registered [ 1293.624206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1293.648441] alg: No test for adler32 (adler32-zlib) [ 1294.575735] LNet: 62736:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1294.599022] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1294.665585] LNet: 62750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.678608] LNetError: 62750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.703080] LNet: Removed LNI 192.168.203.32@tcp [ 1295.219463] Key type .llcrypt unregistered [ 1295.221743] Key type ._llcrypt unregistered [ 1300.619252] Key type ._llcrypt registered [ 1300.625970] Key type .llcrypt registered [ 1300.902816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1300.922388] alg: No test for adler32 (adler32-zlib) [ 1301.817974] LNet: 63009:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1301.826064] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1301.833584] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1301.890404] LNet: 63023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.907583] LNetError: 63023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.911699] LNet: Removed LNI 192.168.203.32@tcp [ 1301.926691] LNet: Removed LNI 10.1.2.3@tcp1 [ 1302.508137] Key type .llcrypt unregistered [ 1302.510179] Key type ._llcrypt unregistered [ 1309.901634] Key type ._llcrypt registered [ 1309.903080] Key type .llcrypt registered [ 1310.038427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1310.046840] alg: No test for adler32 (adler32-zlib) [ 1310.990272] LNet: 63282:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1311.024075] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1311.030352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1311.097165] LNet: 63296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.108111] LNetError: 63296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.111447] LNet: Removed LNI 192.168.203.32@tcp [ 1311.125771] LNet: Removed LNI 10.1.2.3@tcp [ 1311.965822] Key type .llcrypt unregistered [ 1311.968283] Key type ._llcrypt unregistered [ 1319.652871] Key type ._llcrypt registered [ 1319.657813] Key type .llcrypt registered [ 1320.024824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.061400] alg: No test for adler32 (adler32-zlib) [ 1321.118664] LNet: 63557:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1321.141895] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1321.144498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1321.177109] LNet: 63571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.182614] LNetError: 63571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.192936] LNet: Removed LNI 192.168.203.32@tcp [ 1321.211552] LNet: Removed LNI 10.1.2.3@tcp [ 1322.315988] Key type .llcrypt unregistered [ 1322.319273] Key type ._llcrypt unregistered [ 1330.622734] Key type ._llcrypt registered [ 1330.625646] Key type .llcrypt registered [ 1330.929542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.945299] alg: No test for adler32 (adler32-zlib) [ 1331.842637] LNet: 63835:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1331.856173] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1331.859582] LNetError: 63835:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1331.871158] LNetError: Error -100 starting up LNI tcp [ 1331.887833] LNet: Removed LNI 192.168.203.32@tcp [ 1332.682582] Key type .llcrypt unregistered [ 1332.687508] Key type ._llcrypt unregistered [ 1344.960342] Key type ._llcrypt registered [ 1344.962378] Key type .llcrypt registered [ 1346.755273] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:55:43 (1773608143) [ 1347.731456] Key type .llcrypt unregistered [ 1347.735253] Key type ._llcrypt unregistered [ 1354.272802] Key type ._llcrypt registered [ 1354.275438] Key type .llcrypt registered [ 1354.525493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.553136] alg: No test for adler32 (adler32-zlib) [ 1355.568172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.610192] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1363.546173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1364.296354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1364.324628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1364.343701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1364.359257] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1364.556658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1364.582941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1364.607532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.638625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.657450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.678568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.701845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.732792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.747833] LNet: 65092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.770683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.803107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.831454] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1364.833926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.868371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.907402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.954430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.228489] LNet: 65131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.233942] LNetError: 65131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.257620] LNet: Removed LNI 192.168.203.32@tcp [ 1365.675741] Key type .llcrypt unregistered [ 1365.680811] Key type ._llcrypt unregistered [ 1385.650668] Key type ._llcrypt registered [ 1385.658339] Key type .llcrypt registered [ 1385.876902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.896865] alg: No test for adler32 (adler32-zlib) [ 1386.846909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1386.883151] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1395.108297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1405.287977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.331449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1405.372340] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1407.191266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1407.226826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1407.299775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1407.367927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1407.393679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1407.452825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1407.520452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1407.576708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1407.652792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1407.711181] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1407.956479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1407.987569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1408.017789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.045428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.068924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.111800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.152177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.210142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.241786] LNet: 66010:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.257815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.277173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.304703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.353336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.382358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1408.418890] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1408.420752] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1408.433235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1413.156762] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1413.584815] LNet: 66084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.590948] LNetError: 66084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.598723] LNet: Removed LNI 192.168.203.32@tcp [ 1413.629298] LNet: Removed LNI 192.168.203.32@tcp1 [ 1414.303908] Key type .llcrypt unregistered [ 1414.306817] Key type ._llcrypt unregistered [ 1436.065131] Key type ._llcrypt registered [ 1436.066325] Key type .llcrypt registered [ 1437.466495] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:57:14 (1773608234) [ 1438.037162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1438.062668] alg: No test for adler32 (adler32-zlib) [ 1439.152395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.280638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.318255] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1440.525503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.566238] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1441.474215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1441.530677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1441.562644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1441.594397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1441.628669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1441.655814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1441.686581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1441.722294] LNetError: 67185:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1442.276160] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1442.278711] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1444.319303] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1445.191601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1448.418059] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1448.924970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1448.981294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1449.068431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.096241] LNet: 67292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.117231] LNetError: 67292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.144918] LNet: Removed LNI 192.168.203.32@tcp [ 1450.153255] LNetError: 67292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.175591] LNet: Removed LNI 192.168.203.32@tcp1 [ 1450.274397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.381519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.413455] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1451.430657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.459849] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1452.712518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1452.754802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.822644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1452.867483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1452.899903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1452.936813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1452.982443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.011913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1453.025861] LNetError: 67565:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1453.408538] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1456.570416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.554334] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1459.559508] LNet: Skipped 1 previous similar message [ 1464.839239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.886362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.967669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1473.373278] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:57:50 (1773608270) [ 1473.492586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.520968] LNet: 68276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.533914] LNetError: 68276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.548089] LNet: Removed LNI 192.168.203.32@tcp [ 1473.562399] LNet: Removed LNI 192.168.203.32@tcp1 [ 1473.589506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.711105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.751988] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1475.051758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1475.085774] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1476.350978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1476.461993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1476.507333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1476.557922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1476.625769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1476.665413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1476.709752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1476.847678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1477.727289] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.32@tcp [ 1477.727854] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1477.761783] LNet: Skipped 1 previous similar message [ 1491.458632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1491.659333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1491.724493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1491.758377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 500 [ 1491.905691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1497.611541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1497.734240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.780639] LNet: 68757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.792086] LNetError: 68757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.798631] LNetError: 68757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1498.847405] LNet: Removed LNI 192.168.203.32@tcp [ 1498.858583] LNet: Removed LNI 192.168.203.32@tcp1 [ 1498.894282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.026336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.076967] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1499.798125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.838343] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1500.664380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1500.746877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1500.835893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1500.881267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1500.916826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1500.959255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1501.012410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1501.083385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1502.048416] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1502.053237] LNet: Skipped 3 previous similar messages [ 1502.453782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.841749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.901206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.935732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.313796] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:58:38 (1773608318) [ 1521.734396] LNet: 69804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.745516] LNetError: 69804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.758558] LNetError: 69804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1521.767070] LNet: Removed LNI 192.168.203.32@tcp [ 1521.777556] LNet: Removed LNI 192.168.203.32@tcp1 [ 1522.380939] Key type .llcrypt unregistered [ 1522.383185] Key type ._llcrypt unregistered [ 1528.717040] Key type ._llcrypt registered [ 1528.718411] Key type .llcrypt registered [ 1528.927237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.948233] alg: No test for adler32 (adler32-zlib) [ 1529.868259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1529.900832] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1538.958775] Lustre: DEBUG MARKER: Initial discovery [ 1539.051487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1541.566993] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1541.640231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1541.692627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1541.724938] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1541.750238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1541.794710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -l 3 [ 1541.840431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -e local_error [ 1541.904590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m PUT -l 6 [ 1542.175540] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1543.535082] Lustre: DEBUG MARKER: Force oleg332-client.virtnet to discover 192.168.203.132@tcp (in background) [ 1543.597865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1544.895892] Lustre: DEBUG MARKER: Force oleg332-server to discover 192.168.203.32@tcp [ 1546.663070] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1547.047509] Lustre: DEBUG MARKER: Wait for 70562 [ 1548.346331] Lustre: DEBUG MARKER: Finished wait on 70562 [ 1548.475469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1548.506230] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1548.517665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1548.844380] LNet: 71018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.854835] LNetError: 71018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.926621] LNet: Removed LNI 192.168.203.32@tcp [ 1550.272570] Key type .llcrypt unregistered [ 1550.274386] Key type ._llcrypt unregistered [ 1564.621206] Key type ._llcrypt registered [ 1564.623278] Key type .llcrypt registered [ 1565.600503] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:59:23 (1773608363) [ 1566.240689] Key type .llcrypt unregistered [ 1566.243395] Key type ._llcrypt unregistered [ 1571.897530] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1572.024079] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1572.044734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1572.054751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1574.749786] Key type ._llcrypt registered [ 1574.752345] Key type .llcrypt registered [ 1574.969739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1574.988992] alg: No test for adler32 (adler32-zlib) [ 1575.923203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.959456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.978422] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1576.804537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1576.862707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1585.386881] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:59:42 (1773608382) [ 1585.494644] LNet: 73100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.506767] LNetError: 73100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.519601] LNet: Removed LNI 192.168.203.32@tcp [ 1585.547969] LNet: Removed LNI 10.1.2.3@tcp [ 1586.090274] Key type .llcrypt unregistered [ 1586.092890] Key type ._llcrypt unregistered [ 1592.777950] Key type ._llcrypt registered [ 1592.781449] Key type .llcrypt registered [ 1592.970252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.002196] alg: No test for adler32 (adler32-zlib) [ 1593.967262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.031632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.069813] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1594.814443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1594.838469] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1595.984734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.32@tcp [ 1597.025057] LNetError: Unexpected error -22 connecting to 192.168.203.32@tcp at host 192.168.203.32:988 [ 1602.677380] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:59:59 (1773608399) [ 1602.898605] LNet: 74241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.906136] LNetError: 74241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.914635] LNet: Removed LNI 192.168.203.32@tcp [ 1602.928399] LNet: Removed LNI 10.1.2.3@tcp [ 1603.557576] Key type .llcrypt unregistered [ 1603.561773] Key type ._llcrypt unregistered [ 1611.195453] Key type ._llcrypt registered [ 1611.197926] Key type .llcrypt registered [ 1611.378286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.389170] alg: No test for adler32 (adler32-zlib) [ 1612.414332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.503695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1612.528825] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1613.253251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1613.271462] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1614.386060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.32@tcp1 --nid 192.168.203.32@tcp2 [ 1619.193782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp2 [ 1619.219649] LNetError: 75076:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.32@tcp1 to 192.168.203.32@tcp2 - Route table may be misconfigured [ 1619.241522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp2 192.168.203.32@tcp1 [ 1625.458883] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:00:22 (1773608422) [ 1625.552900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.592819] LNet: 75642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.608047] LNetError: 75642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.655745] LNet: Removed LNI 192.168.203.32@tcp1 [ 1626.663035] LNetError: 75642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.683926] LNet: Removed LNI 192.168.203.32@tcp2 [ 1626.741790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.816497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.834400] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1627.448600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1627.469679] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1628.272692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1628.306810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.32@tcp [ 1628.339795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp1 [ 1628.375710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp -e network_timeout [ 1628.412767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -e network_timeout [ 1628.467960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -e network_timeout [ 1628.506909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -e network_timeout [ 1628.553299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1628.580416] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1628.599374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1628.729982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1628.843641] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1628.862468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1635.971873] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:00:32 (1773608432) [ 1636.141375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.170847] LNet: 76508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.204307] LNetError: 76508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.215737] LNet: Removed LNI 192.168.203.32@tcp [ 1638.247265] LNet: Removed LNI 192.168.203.32@tcp1 [ 1638.289232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.439214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1638.847270] LNet: 76548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.295481] Key type .llcrypt unregistered [ 1639.298916] Key type ._llcrypt unregistered [ 1653.196251] Key type ._llcrypt registered [ 1653.198749] Key type .llcrypt registered [ 1654.783255] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:00:51 (1773608451) [ 1655.240912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.270367] alg: No test for adler32 (adler32-zlib) [ 1656.182728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.754833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1659.748462] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:00:56 (1773608456) [ 1660.072510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.104133] LNet: 77647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.129176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.223663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.263959] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1661.219603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1661.242198] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1662.343648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1662.387495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp1 [ 1662.460544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp1 [ 1670.284436] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:01:07 (1773608467) [ 1671.864634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg332-server [ 1673.776658] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:01:10 (1773608470) [ 1675.813749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg332-server [ 1677.572854] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:01:14 (1773608474) [ 1679.510699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg332-server [ 1681.268518] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:01:18 (1773608478) [ 1683.002454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg332-server [ 1684.992796] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:01:21 (1773608481) [ 1687.199912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg332-server [ 1689.252868] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:01:26 (1773608486) [ 1691.417184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg332-server [ 1693.380904] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:01:30 (1773608490) [ 1695.166681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg332-server [ 1697.197201] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:01:34 (1773608494) [ 1699.149604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg332-server [ 1701.190844] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:01:38 (1773608498) [ 1703.303424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg332-server [ 1705.422721] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:01:42 (1773608502) [ 1705.824461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.874913] LNet: 81538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.887401] LNetError: 81538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.895892] LNet: Removed LNI 192.168.203.32@tcp [ 1706.917808] LNet: Removed LNI 192.168.203.32@tcp1 [ 1706.992992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.072238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.098474] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1707.844800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1707.932076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1708.197960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.231966] LNet: 81686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.245098] LNetError: 81686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.257986] LNetError: 81686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1708.272047] LNet: Removed LNI 192.168.203.32@tcp [ 1708.306485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.411826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1708.449993] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1709.275549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1709.404511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.437854] LNetError: 81831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.457207] LNet: Removed LNI 192.168.203.32@tcp [ 1709.470253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.527789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.548214] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1710.186862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1710.232972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1710.429835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.447994] LNet: 81978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.452902] LNet: 81978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1711.530936] LNet: Removed LNI 192.168.203.32@tcp [ 1711.573138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.701223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1711.735179] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1712.632759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1712.930232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.966722] LNetError: 82124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.978631] LNetError: 82124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1713.010111] LNet: Removed LNI 192.168.203.32@tcp [ 1713.045103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.108226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.141815] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1714.027741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1714.098343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1714.343449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.439257] LNet: Removed LNI 192.168.203.32@tcp [ 1715.460075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.530924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1715.581154] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1716.386790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1716.700477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.732324] LNet: 82416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.741551] LNet: 82416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1717.798293] LNet: Removed LNI 192.168.203.32@tcp [ 1717.871593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.982508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.030299] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1718.929430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1719.019818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1719.298318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.319550] LNetError: 82564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.326288] LNetError: 82564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1719.337600] LNet: Removed LNI 192.168.203.32@tcp [ 1719.364706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.419212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1719.464550] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1720.271922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1720.466623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.510028] LNet: Removed LNI 192.168.203.32@tcp [ 1721.556995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.692650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.732890] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1722.440696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1722.546528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1722.824282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.881485] LNet: Removed LNI 192.168.203.32@tcp [ 1723.980122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.070973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1724.105364] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1724.846309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1726.316465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.347659] LNet: 83011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.352828] LNet: 83011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1727.398042] LNet: Removed LNI 192.168.203.32@tcp [ 1727.455470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.528344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.549478] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1728.238174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1728.419959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1729.953926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.968575] LNetError: 83167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.974975] LNetError: 83167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1731.046090] LNet: Removed LNI 192.168.203.32@tcp [ 1731.091281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.178712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1731.204775] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1731.938715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1732.169237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.233847] LNet: Removed LNI 192.168.203.32@tcp [ 1733.271101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.346580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.380504] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1734.007696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1734.066431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.095600] LNet: Removed LNI 192.168.203.32@tcp [ 1734.112498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.155168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.174978] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1734.765599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1742.409160] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:02:19 (1773608539) [ 1742.575490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.604510] LNet: 84147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.609729] LNet: 84147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1742.636223] LNet: Removed LNI 192.168.203.32@tcp [ 1742.670295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.725951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1742.742996] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1744.147109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.187952] LNet: Removed LNI 192.168.203.32@tcp231 [ 1744.224659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.292986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1744.319840] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1744.326948] LNet: 84296:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.794361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.839310] LNet: Removed LNI 192.168.203.32@tcp231 [ 1744.858196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1744.893845] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1745.974934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.003994] LNetError: 84441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.010495] LNetError: 84441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1746.022547] LNet: Removed LNI 192.168.203.32@tcp231 [ 1746.036517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.104325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1746.127473] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1746.132337] LNet: 84449:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.718729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1746.756200] LNet: Removed LNI 192.168.203.32@tcp231 [ 1746.779195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1746.833223] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1748.410316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.475663] LNet: Removed LNI 192.168.203.32@tcp231 [ 1748.496919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.581183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1748.607935] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1748.620671] LNet: 84600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.562973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1749.622348] LNet: Removed LNI 192.168.203.32@tcp231 [ 1749.656046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1749.716558] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1751.739331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.803268] LNet: Removed LNI 192.168.203.32@tcp231 [ 1751.854399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.973508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1752.018297] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1752.030109] LNet: 84751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.754959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1752.814205] LNet: Removed LNI 192.168.203.32@tcp231 [ 1752.840463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1752.879553] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1763.597535] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:02:40 (1773608560) [ 1764.054424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.115721] LNet: Removed LNI 192.168.203.32@tcp231 [ 1764.153442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.284167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1764.355626] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1765.543056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1765.592576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1765.649074] LNet: Removed LNI 192.168.203.32@tcp232 [ 1765.666938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1765.744923] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1766.789757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1766.878944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1766.932847] LNet: Removed LNI 192.168.203.32@tcp232 [ 1766.947423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1766.983958] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1767.948450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1768.007969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1768.058289] LNet: Removed LNI 192.168.203.32@tcp232 [ 1768.087982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1768.120087] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1769.002536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1769.058798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1769.091659] LNet: Removed LNI 192.168.203.32@tcp232 [ 1769.144806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1777.809768] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:02:54 (1773608574) [ 1778.605321] LNet: 86590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.612464] LNet: 86590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1779.535760] Key type .llcrypt unregistered [ 1779.539553] Key type ._llcrypt unregistered [ 1787.509294] Key type ._llcrypt registered [ 1787.511869] Key type .llcrypt registered [ 1787.898395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1787.921547] alg: No test for adler32 (adler32-zlib) [ 1789.177732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1789.253073] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1798.335812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1809.611188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.687302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1809.731407] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1811.484873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1811.553654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1811.608583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1811.642942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1811.680110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1811.721240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1811.768944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1811.828315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1811.863873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1811.893522] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1811.902424] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1812.184310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.260237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1812.337648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.368118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1812.395761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1812.423497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.462655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.505413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.532297] LNet: 87253:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.557420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.596459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.628384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.665461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.704946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.751365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.368468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1813.404977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1813.466454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1813.508405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1813.539461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1813.576309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1813.615236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1813.657509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1813.691678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1813.705946] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1813.968569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.998552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1814.028921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1814.060493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1814.097272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1814.123643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1814.151342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1814.182272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1814.203777] LNet: 87373:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.221432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.261963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.301274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.321515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1814.345757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1814.371190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.822398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1814.861519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1814.891914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1814.919386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1814.948854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1814.981812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1815.016485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1815.050238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1815.084202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1815.102985] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1815.338608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1815.385544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1815.421645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1815.445191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1815.470907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1815.501961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1815.530480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1815.557568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1815.578518] LNet: 87493:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1815.595291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1815.632265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1815.664061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1815.701887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1815.730447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1815.775506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1816.266653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1816.302827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1816.334262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1816.364919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1816.392295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1816.424545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1816.466478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1816.502112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1816.535367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1816.822761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1816.850869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1816.883886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1816.932595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1816.971453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1817.001913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1817.034912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1817.068454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1817.096534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1817.127334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1817.159480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1817.184398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1817.212076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1817.236862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1817.780904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1817.813812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1817.875976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1817.950697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1818.004485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1818.064847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1818.126100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1818.187097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1818.248551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1818.278542] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1818.284129] LNet: Skipped 1 previous similar message [ 1818.559732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1818.588396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1818.615627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1818.648802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1818.691595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1818.718799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1818.747572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1818.781313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1818.802589] LNet: 87734:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1818.811525] LNet: 87734:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1818.831945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1818.867225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1818.919445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1818.974995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.029336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.094903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1819.753804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1819.812383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1819.844773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1819.894242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1819.932284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1819.976235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1820.033509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1820.073319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1820.117303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1820.141175] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1820.408399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1820.462812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1820.532873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1820.578746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1820.612624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1820.658399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1820.714940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1820.757837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1820.806107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1820.873509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1820.915623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1820.961307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1821.008466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1821.044389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1827.403242] LNet: 87928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.413561] LNetError: 87928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.419248] LNet: Removed LNI 192.168.203.32@tcp [ 1827.437000] LNet: Removed LNI 192.168.203.32@tcp1 [ 1828.036580] Key type .llcrypt unregistered [ 1828.050222] Key type ._llcrypt unregistered [ 1844.773848] Key type ._llcrypt registered [ 1844.777215] Key type .llcrypt registered [ 1846.393394] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:04:03 (1773608643) [ 1846.783406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.800843] alg: No test for adler32 (adler32-zlib) [ 1847.756692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.812815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.835707] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1848.667798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1848.690833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1848.709474] LNetError: 88908:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1848.804799] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1848.829313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.32@tcp [ 1848.941686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1856.093267] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:04:12 (1773608652) [ 1856.202285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.262483] LNet: 89489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.279436] LNetError: 89489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.304990] LNet: Removed LNI 192.168.203.32@tcp [ 1856.407584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.489633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.516113] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1857.483611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1857.505071] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1858.833794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.32@tcp [ 1858.864345] LNetError: 89763:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1859.555597] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1860.210736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.32@tcp [ 1869.982554] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:04:27 (1773608667) [ 1871.717845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg332-server [ 1873.706940] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:04:30 (1773608670) [ 1876.026889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg332-server [ 1877.851666] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:04:34 (1773608674) [ 1880.354419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg332-server [ 1882.468520] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:04:39 (1773608679) [ 1882.592865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.609689] LNet: 91393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.615926] LNetError: 91393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.679184] LNet: Removed LNI 192.168.203.32@tcp [ 1883.681109] LNetError: 91393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.710054] LNet: Removed LNI 192.168.203.32@tcp1 [ 1883.797673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.912857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1883.948424] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1884.809790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1884.928782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1884.994662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1885.156521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1885.250442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1885.326509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1893.108700] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:04:50 (1773608690) [ 1893.366725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.421730] LNet: 92120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.430739] LNetError: 92120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.457479] LNet: Removed LNI 192.168.203.32@tcp [ 1893.491409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.561890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1893.588564] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1901.438591] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:04:58 (1773608698) [ 1901.580670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.634622] LNet: 92819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.649996] LNetError: 92819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.702459] LNet: Removed LNI 192.168.203.32@tcp [ 1901.755511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.816657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.835246] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1908.761257] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:05:05 (1773608705) [ 1910.592544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1912.214121] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:05:09 (1773608709) [ 1912.930901] LNet: 93858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.943207] LNetError: 93858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.957944] LNet: Removed LNI 192.168.203.32@tcp [ 1913.663811] Key type .llcrypt unregistered [ 1913.668484] Key type ._llcrypt unregistered [ 1920.794252] Key type ._llcrypt registered [ 1920.798507] Key type .llcrypt registered [ 1920.982754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.002267] alg: No test for adler32 (adler32-zlib) [ 1921.970324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1921.998944] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1929.856205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1935.010612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.132@tcp [ 1935.341300] LNet: 94276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.345583] LNetError: 94276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.354794] LNet: Removed LNI 192.168.203.32@tcp [ 1935.763409] Key type .llcrypt unregistered [ 1935.775555] Key type ._llcrypt unregistered [ 1948.584621] Key type ._llcrypt registered [ 1948.586571] Key type .llcrypt registered [ 1949.871747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1951.314553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1953.321624] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:05:50 (1773608750) [ 1953.682243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.700326] alg: No test for adler32 (adler32-zlib) [ 1954.706333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.826725] LNet: 95362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.378443] Key type .llcrypt unregistered [ 1955.384113] Key type ._llcrypt unregistered [ 1961.291191] Key type ._llcrypt registered [ 1961.295292] Key type .llcrypt registered [ 1961.535757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.560326] alg: No test for adler32 (adler32-zlib) [ 1962.612874] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1962.616402] LNet: 95619:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1962.627663] LNet: 95619:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1962.670342] LNetError: 95631:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 1962.689525] LNetError: 95631:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.33@tcp has gone from up to down [ 1963.443490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.477568] LNet: 95677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.494583] LNetError: 95677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.528934] LNet: Removed LNI 192.168.203.32@tcp [ 1963.588376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.658972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1963.710144] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1963.714924] LNet: 95684:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1963.725506] LNet: 95684:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1963.735271] LNetError: 95684:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.203.33@tcp is a gateway. Can not delete it [ 1963.740946] LNetError: 95684:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.203.33@tcp is a router. Can not be deleted [ 1963.755629] LNet: 95684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1963.795209] LNetError: 95681:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 1963.817867] LNetError: 95681:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1963.831842] LNetError: 95681:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.34@tcp has gone from up to down [ 1963.855344] LNetError: 95681:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1970.133994] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:06:07 (1773608767) [ 1972.190558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg332-server [ 1974.496684] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:06:11 (1773608771) [ 1977.137872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg332-server [ 1979.026391] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:06:16 (1773608776) [ 1981.683589] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1983.679245] Key type lgssc registered [ 1985.644424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.482237] Key type lgssc unregistered [ 2008.956426] LNet: 98107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.961669] LNetError: 98107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.977654] LNet: Removed LNI 192.168.203.32@tcp [ 2009.988784] Key type .llcrypt unregistered [ 2009.999756] Key type ._llcrypt unregistered [ 2024.012412] Key type ._llcrypt registered [ 2024.014023] Key type .llcrypt registered [ 2025.309766] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:07:02 (1773608822) [ 2025.743604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2025.759722] alg: No test for adler32 (adler32-zlib) [ 2026.695850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.755984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2026.822908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2026.875454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2026.939499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.515779] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:07:11 (1773608831) [ 2035.641696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.716112] LNet: 99509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.755850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.853955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.901759] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2037.266115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.290449] LNet: 99659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.296931] LNetError: 99659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.323498] LNet: Removed LNI 192.168.203.32@tcp [ 2037.384502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.442172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2037.455962] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2038.310453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.361883] LNet: 99689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.371319] LNetError: 99689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.391774] LNet: Removed LNI 192.168.203.32@tcp [ 2038.446483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.561972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2038.615520] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2047.435283] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:07:23 (1773608843) [ 2047.597174] LNet: 100272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.609135] LNetError: 100272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.634984] LNet: Removed LNI 192.168.203.32@tcp [ 2048.627789] Key type .llcrypt unregistered [ 2048.636747] Key type ._llcrypt unregistered [ 2056.120310] Key type ._llcrypt registered [ 2056.121777] Key type .llcrypt registered [ 2056.447485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2056.466413] alg: No test for adler32 (adler32-zlib) [ 2057.503425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.622425] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2057.626594] LNet: *** cfs_fail_loc=e003, val=100*** [ 2058.546246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2058.567925] LNet: *** cfs_fail_loc=e003, val=100*** [ 2058.571152] LNet: Skipped 2 previous similar messages [ 2058.583988] Ping reply received with mapping: [ 2058.586642] found metadata reply NI[192.168.203.32@tcp] [ 2058.590211] 20 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2058.592451] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2058.595262] 5a 5a 5a 5a 5a 5a 5a 5a [ 2058.734225] LNet: 100678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.739725] LNetError: 100678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.750995] LNet: Removed LNI 192.168.203.32@tcp [ 2059.354590] Key type .llcrypt unregistered [ 2059.357104] Key type ._llcrypt unregistered [ 2070.985500] Key type ._llcrypt registered [ 2070.987267] Key type .llcrypt registered [ 2072.185844] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:07:49 (1773608869) [ 2072.729773] Key type .llcrypt unregistered [ 2072.731422] Key type ._llcrypt unregistered [ 2080.313553] Key type ._llcrypt registered [ 2080.317678] Key type .llcrypt registered [ 2080.609929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2080.630127] alg: No test for adler32 (adler32-zlib) [ 2081.996628] LNetError: 101724:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2082.487489] Key type .llcrypt unregistered [ 2082.489087] Key type ._llcrypt unregistered [ 2088.681803] Key type ._llcrypt registered [ 2088.683360] Key type .llcrypt registered [ 2089.945406] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:08:07 (1773608887) [ 2091.098328] Key type .llcrypt unregistered [ 2091.100235] Key type ._llcrypt unregistered [ 2097.539750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2097.552266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2100.374787] Key type ._llcrypt registered [ 2100.377535] Key type .llcrypt registered [ 2100.609203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2100.618569] alg: No test for adler32 (adler32-zlib) [ 2101.572364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.657645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2111.723240] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:08:28 (1773608908) [ 2111.998910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.056305] LNet: 103505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.073469] LNetError: 103505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2112.108256] LNet: Removed LNI 10.1.2.3@tcp [ 2112.150923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.132274] Key type .llcrypt unregistered [ 2113.133659] Key type ._llcrypt unregistered [ 2120.631607] Key type ._llcrypt registered [ 2120.635583] Key type .llcrypt registered [ 2120.817594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.831923] alg: No test for adler32 (adler32-zlib) [ 2121.797260] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2121.896317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2122.122054] LNet: 103794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.135925] LNetError: 103794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.158586] LNet: Removed LNI 192.168.203.32@tcp [ 2122.700713] Key type .llcrypt unregistered [ 2122.705696] Key type ._llcrypt unregistered [ 2135.161545] Key type ._llcrypt registered [ 2135.165661] Key type .llcrypt registered [ 2137.340254] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:08:53 (1773608933) [ 2137.925304] Key type .llcrypt unregistered [ 2137.927894] Key type ._llcrypt unregistered [ 2144.189607] Key type ._llcrypt registered [ 2144.204516] Key type .llcrypt registered [ 2144.611669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2144.637367] alg: No test for adler32 (adler32-zlib) [ 2153.737391] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:09:10 (1773608950) [ 2153.896780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.957114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.065442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.095692] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2154.559899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.585087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2154.607902] LNet: Removed LNI 192.168.203.32@tcp [ 2154.832850] LNet: 105538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.120725] Key type .llcrypt unregistered [ 2155.123877] Key type ._llcrypt unregistered [ 2166.287482] Key type ._llcrypt registered [ 2166.290318] Key type .llcrypt registered [ 2167.500373] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:09:24 (1773608964) [ 2167.980103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.003710] alg: No test for adler32 (adler32-zlib) [ 2168.884903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.925844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.943226] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2169.560859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2169.591263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.32@tcp [ 2169.611356] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2169.615421] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2169.619338] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2169, 0, 0 [ 2169.625781] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2169.629448] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2169.633366] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2169.637915] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2169, 0, 0 [ 2169.642978] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2169.645267] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2169.647606] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2169.650627] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2169, 0, 2219 [ 2169.656880] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2169.659024] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2169.662487] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2169.665073] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2169, 0, 2219 [ 2169.669180] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2176.633028] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:09:33 (1773608973) [ 2177.168573] LNet: 107083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.173556] LNetError: 107083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.186577] LNet: Removed LNI 192.168.203.32@tcp [ 2177.835616] Key type .llcrypt unregistered [ 2177.839693] Key type ._llcrypt unregistered [ 2184.971688] Key type ._llcrypt registered [ 2184.973305] Key type .llcrypt registered [ 2185.129963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2185.165267] alg: No test for adler32 (adler32-zlib) [ 2186.126996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2186.152773] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2194.300942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 2203.761497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.819636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2203.838794] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 2205.032829] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2205.227806] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2205.244084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2205.273395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2207.793808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2207.819942] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2220.311378] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:10:17 (1773609017) [ 2220.516165] LNet: 108594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.523576] LNetError: 108594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.535173] LNet: Removed LNI 192.168.203.32@tcp [ 2221.538888] LNetError: 108594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.556371] LNet: Removed LNI 192.168.203.32@tcp1 [ 2221.580093] LNet: Removed LNI 10.1.2.3@tcp99 [ 2222.467640] Key type .llcrypt unregistered [ 2222.473706] Key type ._llcrypt unregistered [ 2229.049253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2229.070181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2231.576860] Key type ._llcrypt registered [ 2231.579991] Key type .llcrypt registered [ 2231.864916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2231.898472] alg: No test for adler32 (adler32-zlib) [ 2232.876597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.936951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.964362] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2233.543248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2233.565818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2235.723980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2235.781206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2235.825846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2235.848280] LNetError: 109339:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2235.871642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2235.923387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2236.422336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2236.505449] LNet: 109359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.510856] LNetError: 109359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.515799] LNet: Removed LNI 192.168.203.32@tcp [ 2236.529043] LNet: Removed LNI 10.1.2.3@tcp [ 2237.530842] Key type .llcrypt unregistered [ 2237.532995] Key type ._llcrypt unregistered [ 2250.793919] Key type ._llcrypt registered [ 2250.798379] Key type .llcrypt registered [ 2252.572724] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:10:49 (1773609049) [ 2253.010917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.030713] alg: No test for adler32 (adler32-zlib) [ 2253.976871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.075898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.138953] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2254.889824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2254.992462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg332-client@tcp [ 2262.331720] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:10:59 (1773609059) [ 2262.450666] LNet: 110886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.456634] LNetError: 110886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.496405] LNet: Removed LNI 192.168.203.32@tcp [ 2263.587879] Key type .llcrypt unregistered [ 2263.592229] Key type ._llcrypt unregistered [ 2269.884990] Key type ._llcrypt registered [ 2269.887054] Key type .llcrypt registered [ 2270.040817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.064824] alg: No test for adler32 (adler32-zlib) [ 2270.953299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2270.983696] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2271.019698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2276.600428] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:11:14 (1773609074) [ 2278.512722] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2280.106447] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:11:17 (1773609077) [ 2281.884412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2283.474285] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:11:20 (1773609080) [ 2283.655695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.705587] LNet: 112313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.714715] LNetError: 112313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.730486] LNet: Removed LNI 192.168.203.32@tcp [ 2283.750737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.828276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2283.853555] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2284.625933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2284.733402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2290.434858] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:11:28 (1773609088) [ 2290.755895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.786273] LNet: 113022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.790680] LNetError: 113022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.818804] LNet: Removed LNI 192.168.203.32@tcp [ 2290.842750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.910973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2290.937200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2291.590576] Key type .llcrypt unregistered [ 2291.592792] Key type ._llcrypt unregistered [ 2301.572524] Key type ._llcrypt registered [ 2301.574662] Key type .llcrypt registered [ 2302.586412] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:11:39 (1773609099) [ 2302.979577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.992783] alg: No test for adler32 (adler32-zlib) [ 2303.963322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.035844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.081602] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2304.823847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2304.909890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2312.507659] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:11:49 (1773609109) [ 2312.662276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.691856] LNet: 114574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.705514] LNetError: 114574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.733644] LNet: Removed LNI 192.168.203.32@tcp [ 2312.800619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.910525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2312.974827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2319.498471] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:11:56 (1773609116) [ 2319.587647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.612452] LNet: 115152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.637246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.735857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2320.213958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2320.253649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2320.321223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2320.853969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2320.885579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2320.995087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.027622] LNet: 115193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.068667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.138649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2321.731222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2321.765616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2321.831894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2322.228837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2322.264823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2328.935268] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:12:05 (1773609125) [ 2329.051097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.072585] LNet: 115785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.112686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.195449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.250958] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2330.122905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2330.148039] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2331.034933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2331.113623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2331.406163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.462737] LNetError: 116073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.475502] LNet: Removed LNI 192.168.203.32@tcp [ 2331.499969] LNet: Removed LNI 192.168.203.32@tcp2 [ 2331.532875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.609778] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2331.621942] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2331.637548] LNet: 116079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2332.433113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2332.688853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.710834] LNetError: 116116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.714099] LNetError: 116116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2332.723989] LNet: Removed LNI 192.168.203.32@tcp [ 2332.737350] LNet: Removed LNI 192.168.203.32@tcp2 [ 2332.759889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.829924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2332.874458] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2333.555138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2333.579389] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2334.392964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2334.439500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2334.508614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2334.782861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.801606] LNet: 116400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.805699] LNet: 116400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2334.810945] LNetError: 116400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.816613] LNetError: 116400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2334.824090] LNet: Removed LNI 192.168.203.32@tcp [ 2335.846546] LNet: Removed LNI 192.168.203.32@tcp2 [ 2335.902868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.996290] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2336.000776] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2336.004349] LNet: 116407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.518652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2336.711802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.730872] LNet: Removed LNI 192.168.203.32@tcp [ 2337.776606] LNet: Removed LNI 192.168.203.32@tcp2 [ 2337.847831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.903723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.921684] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2338.554835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.574707] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2339.563174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2339.648966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2339.857849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.882561] LNetError: 116723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.885837] LNetError: 116723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2340.895223] LNet: Removed LNI 192.168.203.32@tcp [ 2340.906182] LNet: Removed LNI 192.168.203.32@tcp2 [ 2340.971896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.066444] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2341.076321] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2341.084325] LNet: 116730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.751600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2341.979938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.071129] LNet: Removed LNI 192.168.203.32@tcp [ 2343.083185] LNet: Removed LNI 192.168.203.32@tcp2 [ 2343.135791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.219960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.293684] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2344.106829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2344.123497] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2345.153704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2345.190482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2345.259558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2345.526987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.545957] LNet: 117052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.549766] LNet: 117052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2346.591143] LNet: Removed LNI 192.168.203.32@tcp [ 2347.626703] LNet: Removed LNI 192.168.203.32@tcp2 [ 2347.680944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.745712] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2347.754951] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2347.759489] LNet: 117058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.203790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2348.400641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.415564] LNetError: 117095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.421591] LNetError: 117095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2348.426557] LNet: Removed LNI 192.168.203.32@tcp [ 2349.485759] LNet: Removed LNI 192.168.203.32@tcp2 [ 2349.525916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.604504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.631451] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2350.318912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.331956] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2351.028554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2351.081619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2351.150230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2351.467315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.544068] LNet: Removed LNI 192.168.203.32@tcp [ 2352.561437] LNet: Removed LNI 192.168.203.32@tcp2 [ 2352.585559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.638203] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2352.646834] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2352.650511] LNet: 117398:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.086723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2353.383320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.463785] LNet: Removed LNI 192.168.203.32@tcp [ 2354.478328] LNet: Removed LNI 192.168.203.32@tcp2 [ 2354.504992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.603389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.626419] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2355.401627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.418989] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2356.261719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2356.302191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2356.334610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2356.397577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2356.736609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.791208] LNet: Removed LNI 192.168.203.32@tcp [ 2358.821748] LNet: Removed LNI 192.168.203.32@tcp2 [ 2358.885912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.961762] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2358.969652] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2359.725377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2367.256745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2369.207885] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:12:46 (1773609166) [ 2369.509573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.528247] LNet: 118519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.540546] LNet: 118519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2369.559403] LNetError: 118519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.573382] LNetError: 118519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2369.583411] LNet: Removed LNI 192.168.203.32@tcp [ 2370.676401] LNet: Removed LNI 192.168.203.32@tcp2 [ 2370.723462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.109358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.143252] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2371.843457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2371.859500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2372.737811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.768387] LNet: Removed LNI 192.168.203.32@tcp [ 2372.781223] LNet: Removed LNI 10.1.2.3@tcp [ 2372.802674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.857789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2372.875962] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2372.884024] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2373.256893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2373.282169] LNet: Removed LNI 192.168.203.32@tcp [ 2373.295539] LNet: Removed LNI 10.1.2.3@tcp [ 2380.111275] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:12:56 (1773609176) [ 2380.497086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.534816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.633292] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2380.761143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.770264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2380.780124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2383.381822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.444626] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2384.677509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2384.706376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2387.275952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2387.311827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2387.372436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.32@tcp -m PUT -e local_timeout -r 1 [ 2387.436189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2387.493881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2387.501060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2387.736903] LNetError: 119459:0:(lib-move.c:4969:lnet_parse()) 192.168.203.32@tcp, src 192.168.203.32@tcp: Dropping PUT (error -108 looking up sender) [ 2389.791554] LNet: Removed LNI 192.168.203.32@tcp [ 2389.805899] LNet: Removed LNI 10.1.2.3@tcp [ 2390.406183] Key type .llcrypt unregistered [ 2390.408592] Key type ._llcrypt unregistered [ 2402.165335] Key type ._llcrypt registered [ 2402.167498] Key type .llcrypt registered [ 2403.209996] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:13:20 (1773609200) [ 2403.616397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.634900] alg: No test for adler32 (adler32-zlib) [ 2404.551629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.850783] LNet: 120770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.500640] Key type .llcrypt unregistered [ 2405.502057] Key type ._llcrypt unregistered [ 2410.950735] Key type ._llcrypt registered [ 2410.955344] Key type .llcrypt registered [ 2411.121698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2411.144678] alg: No test for adler32 (adler32-zlib) [ 2412.074798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2412.102269] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2418.366251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 2418.552601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2419.465950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 2419.707759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2419.760593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2419.793657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2419.818574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.853692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.881547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.908394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.951222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.986091] LNet: 121219:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.009605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.058487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.125125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.177167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.226174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.292116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2421.008406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2421.344454] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 2421.380657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2421.422124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2421.470463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2421.493929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2421.519072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.555614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.594419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.618813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.632870] LNet: 121337:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.651043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.672740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.699309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.724650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.757808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.787630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.325641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 2422.559864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.600592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2422.625499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.649514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.689560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.717623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.745102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.787720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.810173] LNet: 121454:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.829216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.856394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.887670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.916826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.949495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.979680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.701651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2423.997781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.029874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2424.076342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.123726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.164805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.198925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.245464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.295843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.360832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.415128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.470079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.571022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.677766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.727779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.169251] LNet: 121617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.179845] LNetError: 121617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.225041] LNet: Removed LNI 192.168.203.32@tcp [ 2426.771764] Key type .llcrypt unregistered [ 2426.775280] Key type ._llcrypt unregistered [ 2445.638592] Key type ._llcrypt registered [ 2445.640960] Key type .llcrypt registered [ 2447.054980] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:14:04 (1773609244) [ 2447.671790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.699069] alg: No test for adler32 (adler32-zlib) [ 2448.580944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.268850] LNet: 122504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.765571] Key type .llcrypt unregistered [ 2449.767957] Key type ._llcrypt unregistered [ 2456.211839] Key type ._llcrypt registered [ 2456.219655] Key type .llcrypt registered [ 2456.497396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2456.533398] alg: No test for adler32 (adler32-zlib) [ 2457.435660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2457.464878] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2466.159873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 2475.954221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.983729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2476.009880] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 2476.955104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2477.773191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 2477.983445] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 2478.097263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2478.130481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2478.161826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2478.197504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2478.269656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2478.349469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2478.422026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2478.457388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2478.477790] LNet: 123157:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.492859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2478.590558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2478.641339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2478.689861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2478.749890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2478.795510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2479.458611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2479.681047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.711970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2479.751590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.794755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.833151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.887759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.921708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.962838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.995205] LNet: 123274:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2480.020596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.064944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.111219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.154320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.190818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.229973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.755815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 2481.048839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.092120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2481.132592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.185976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.235635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.277291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.311247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.349540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.383409] LNet: 123391:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.411544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.464629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.515797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.556745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.593209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2481.620890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.235600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2482.544143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2482.589461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2482.646727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2482.702908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2482.763786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2482.814925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2482.863788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2482.914637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2482.961875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.003538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.057116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2483.096985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2483.152872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2483.234910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.202119] LNet: 123589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.211521] LNetError: 123589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.224221] LNet: Removed LNI 192.168.203.32@tcp [ 2488.245538] LNet: Removed LNI 192.168.203.32@tcp1 [ 2488.705674] Key type .llcrypt unregistered [ 2488.708311] Key type ._llcrypt unregistered [ 2506.246920] Key type ._llcrypt registered [ 2506.251366] Key type .llcrypt registered [ 2508.040630] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:15:04 (1773609304) [ 2516.065132] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:15:13 (1773609313) [ 2524.695333] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:15:21 (1773609321) [ 2532.116253] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:15:29 (1773609329) [ 2540.143659] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:15:37 (1773609337) [ 2547.820335] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:15:44 (1773609344) [ 2556.279031] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:15:53 (1773609353) [ 2563.670533] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:16:00 (1773609360) [ 2573.466601] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:16:10 (1773609370) [ 2581.849499] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:16:18 (1773609378) [ 2589.355548] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:16:26 (1773609386) [ 2598.881542] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:16:35 (1773609395) [ 2606.430123] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:16:43 (1773609403) [ 2614.913439] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:16:51 (1773609411) [ 2623.484765] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:17:00 (1773609420) [ 2630.796194] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:17:08 (1773609428) [ 2639.813298] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:17:16 (1773609436) [ 2648.061643] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:17:25 (1773609445) [ 2648.716672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2648.730817] alg: No test for adler32 (adler32-zlib) [ 2649.755923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2649.887590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2649.920630] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2651.332498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2651.366603] LNet: 134302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2651.371946] LNetError: 134302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2651.411212] LNet: Removed LNI 192.168.203.32@tcp [ 2651.453565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2651.569972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2651.616819] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2660.032818] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:17:37 (1773609457) [ 2668.662109] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:17:45 (1773609465) [ 2677.025274] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:17:54 (1773609474) [ 2685.774956] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:18:02 (1773609482) [ 2693.449117] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:18:10 (1773609490) [ 2702.061691] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:18:18 (1773609498) [ 2710.510155] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:18:27 (1773609507) [ 2719.443412] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:18:36 (1773609516) [ 2727.754754] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:18:44 (1773609524) [ 2735.498328] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:18:52 (1773609532) [ 2743.283190] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:19:00 (1773609540) [ 2750.313874] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:19:07 (1773609547) [ 2758.704416] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2543 sec ========== 17:19:15 (1773609555) [ 2759.054064] LNet: 141769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2759.061446] LNetError: 141769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2759.084138] LNet: Removed LNI 192.168.203.32@tcp [ 2759.917823] Key type .llcrypt unregistered [ 2759.919627] Key type ._llcrypt unregistered [ 2772.325862] Key type ._llcrypt registered [ 2772.336528] Key type .llcrypt registered [ 2773.104634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2773.114381] alg: No test for adler32 (adler32-zlib) [ 2774.264990] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2774.508374] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2776.287347] Key type lgssc registered [ 2778.026072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2846.244210] Lustre: Mounted lustre-client [ 2853.466727] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2867.614238] LustreError: 144232:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8a11afa56800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2867.647733] Lustre: Unmounted lustre-client [ 2890.245802] Key type lgssc unregistered [ 2890.782843] LNet: 144715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.805811] LNetError: 144715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.825295] LNet: Removed LNI 192.168.203.32@tcp [ 2892.015443] Key type .llcrypt unregistered [ 2892.017843] Key type ._llcrypt unregistered