[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 414563627 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001007] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006282] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009085] LSM: Security Framework initializing [ 0.010023] Yama: becoming mindful. [ 0.010781] SELinux: Initializing. [ 0.011035] *** VALIDATE selinux *** [ 0.017460] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021615] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023066] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024083] *** VALIDATE tmpfs *** [ 0.025371] *** VALIDATE proc *** [ 0.026000] *** VALIDATE cgroup *** [ 0.026000] *** VALIDATE cgroup2 *** [ 0.026000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.026115] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.027004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.028024] Spectre V2 : User space: Vulnerable [ 0.029005] Speculative Store Bypass: Vulnerable [ 0.032059] debug: unmapping init [mem 0xffffffffa6859000-0xffffffffa6860fff] [ 0.034303] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036210] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037016] ... version: 2 [ 0.037896] ... bit width: 48 [ 0.038006] ... generic registers: 4 [ 0.038893] ... value mask: 0000ffffffffffff [ 0.039009] ... max period: 00007fffffffffff [ 0.040007] ... fixed-purpose events: 3 [ 0.040816] ... event mask: 000000070000000f [ 0.041255] rcu: Hierarchical SRCU implementation. [ 0.043148] smp: Bringing up secondary CPUs ... [ 0.044447] x86: Booting SMP configuration: [ 0.045020] .... node #0, CPUs: #1 #2 #3 [ 0.047533] smp: Brought up 1 node, 4 CPUs [ 0.049009] smpboot: Max logical packages: 1 [ 0.049948] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128013] node 0 deferred pages initialised in 77ms [ 0.130577] devtmpfs: initialized [ 0.130895] x86/mm: Memory block size: 128MB [ 0.133092] gcov: version magic: 0x41383552 [ 0.134000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136183] pinctrl core: initialized pinctrl subsystem [ 0.138080] [ 0.138424] ************************************************************* [ 0.139005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140005] ** ** [ 0.142007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143006] ** ** [ 0.145008] ** This means that this kernel is built to expose internal ** [ 0.147005] ** IOMMU data structures, which may compromise security on ** [ 0.148004] ** your system. ** [ 0.150007] ** ** [ 0.152011] ** If you see this message and you are not debugging the ** [ 0.154010] ** kernel, report this immediately to your vendor! ** [ 0.156008] ** ** [ 0.159010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161007] ************************************************************* [ 0.164638] NET: Registered protocol family 16 [ 0.166393] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176020] cpuidle: using governor menu [ 0.177962] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181632] PCI: Using configuration type 1 for base access [ 0.184131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194152] cryptd: max_cpu_qlen set to 1000 [ 0.196188] ACPI: Added _OSI(Module Device) [ 0.197010] ACPI: Added _OSI(Processor Device) [ 0.199009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204316] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209561] ACPI: Interpreter enabled [ 0.211049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212007] ACPI: Using IOAPIC for interrupt routing [ 0.218133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223497] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234186] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245323] acpiphp: Slot [2] registered [ 0.247119] acpiphp: Slot [5] registered [ 0.248118] acpiphp: Slot [6] registered [ 0.249075] acpiphp: Slot [3] registered [ 0.250069] acpiphp: Slot [4] registered [ 0.251115] acpiphp: Slot [7] registered [ 0.253068] acpiphp: Slot [8] registered [ 0.254066] acpiphp: Slot [9] registered [ 0.255059] acpiphp: Slot [10] registered [ 0.256122] acpiphp: Slot [11] registered [ 0.258063] acpiphp: Slot [12] registered [ 0.259000] acpiphp: Slot [13] registered [ 0.259000] acpiphp: Slot [14] registered [ 0.259000] acpiphp: Slot [15] registered [ 0.259000] acpiphp: Slot [16] registered [ 0.259000] acpiphp: Slot [17] registered [ 0.262070] acpiphp: Slot [18] registered [ 0.263070] acpiphp: Slot [19] registered [ 0.264062] acpiphp: Slot [20] registered [ 0.265000] acpiphp: Slot [21] registered [ 0.265000] acpiphp: Slot [22] registered [ 0.266060] acpiphp: Slot [23] registered [ 0.267074] acpiphp: Slot [24] registered [ 0.268074] acpiphp: Slot [25] registered [ 0.269067] acpiphp: Slot [26] registered [ 0.271059] acpiphp: Slot [27] registered [ 0.272058] acpiphp: Slot [28] registered [ 0.273065] acpiphp: Slot [29] registered [ 0.274079] acpiphp: Slot [30] registered [ 0.275070] acpiphp: Slot [31] registered [ 0.276042] PCI host bridge to bus 0000:00 [ 0.277012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.279011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288341] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291205] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294323] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299948] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.303415] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313709] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316762] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324568] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.337011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.345363] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.348009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.356011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.367012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.374149] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.378011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.382011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.391014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.398443] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.401431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.402288] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.403319] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.405246] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.411117] iommu: Default domain type: Passthrough [ 0.413259] SCSI subsystem initialized [ 0.414082] ACPI: bus type USB registered [ 0.415077] usbcore: registered new interface driver usbfs [ 0.416037] usbcore: registered new interface driver hub [ 0.418049] usbcore: registered new device driver usb [ 0.419222] pps_core: LinuxPPS API ver. 1 registered [ 0.421008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.423036] PTP clock support registered [ 0.424249] EDAC MC: Ver: 3.0.0 [ 0.426083] PCI: Using ACPI for IRQ routing [ 0.427571] NetLabel: Initializing [ 0.428006] NetLabel: domain hash size = 128 [ 0.429005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.431059] NetLabel: unlabeled traffic allowed by default [ 0.433003] vgaarb: loaded [ 0.434280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.436008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.442397] clocksource: Switched to clocksource kvm-clock [ 0.552163] VFS: Disk quotas dquot_6.6.0 [ 0.553673] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.555307] *** VALIDATE ramfs *** [ 0.556158] *** VALIDATE hugetlbfs *** [ 0.557181] pnp: PnP ACPI init [ 0.558900] pnp: PnP ACPI: found 6 devices [ 0.574225] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.576527] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.577925] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.579322] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.580817] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.582386] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.584316] NET: Registered protocol family 2 [ 0.586285] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.589741] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.592142] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.596045] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.598400] TCP: Hash tables configured (established 65536 bind 65536) [ 0.600416] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.602595] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.604609] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.606727] NET: Registered protocol family 1 [ 0.609588] RPC: Registered named UNIX socket transport module. [ 0.611090] RPC: Registered udp transport module. [ 0.612253] RPC: Registered tcp transport module. [ 0.613655] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.615429] NET: Registered protocol family 44 [ 0.616723] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.618423] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.619992] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.621855] PCI: CLS 0 bytes, default 64 [ 0.623301] Unpacking initramfs... [ 2.192145] debug: unmapping init [mem 0xffff9d3b7cc64000-0xffff9d3b7ffcffff] [ 2.196802] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.198901] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.201692] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.743446] Initialise system trusted keyrings [ 2.745032] Key type blacklist registered [ 2.748409] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756314] zbud: loaded [ 2.758602] *** VALIDATE nfs *** [ 2.759762] *** VALIDATE nfs4 *** [ 2.760878] pstore: using deflate compression [ 2.764691] Platform Keyring initialized [ 2.894162] NET: Registered protocol family 38 [ 2.895414] Key type asymmetric registered [ 2.896527] Asymmetric key parser 'x509' registered [ 2.897804] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.900367] io scheduler mq-deadline registered [ 2.901540] io scheduler kyber registered [ 2.902736] io scheduler bfq registered [ 2.952512] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.957957] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.961939] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.965684] ACPI: Power Button [PWRF] [ 2.975733] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.984355] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.999087] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.028544] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.058461] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.065810] Non-volatile memory driver v1.3 [ 3.067422] Linux agpgart interface v0.103 [ 3.125618] virtio_blk virtio1: [vda] 134728 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.132878] vda: detected capacity change from 0 to 68980736 [ 3.160945] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.164817] vdb: detected capacity change from 0 to 1073741824 [ 3.173804] libphy: Fixed MDIO Bus: probed [ 3.179763] usbcore: registered new interface driver usbserial_generic [ 3.183049] usbserial: USB Serial support registered for generic [ 3.185386] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.191310] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.193358] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.196276] mousedev: PS/2 mouse device common for all mice [ 3.199338] rtc_cmos 00:05: RTC can wake from S4 [ 3.202830] rtc_cmos 00:05: registered as rtc0 [ 3.205175] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.208717] intel_pstate: CPU model not supported [ 3.212599] hid: raw HID events driver (C) Jiri Kosina [ 3.215052] usbcore: registered new interface driver usbhid [ 3.218182] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.219570] usbhid: USB HID core driver [ 3.219737] drop_monitor: Initializing network drop monitor service [ 3.219885] Initializing XFRM netlink socket [ 3.220854] NET: Registered protocol family 10 [ 3.230177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.236113] Segment Routing with IPv6 [ 3.238379] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.244511] NET: Registered protocol family 17 [ 3.245029] mpls_gso: MPLS GSO support [ 3.258815] RAS: Correctable Errors collector initialized. [ 3.261066] AVX version of gcm_enc/dec engaged. [ 3.262958] AES CTR mode by8 optimization enabled [ 3.354644] sched_clock: Marking stable (3354566147, 0)->(4064988763, -710422616) [ 3.361310] registered taskstats version 1 [ 3.363404] Loading compiled-in X.509 certificates [ 3.365522] zswap: loaded using pool lzo/zbud [ 3.447692] Key type big_key registered [ 3.464676] Key type encrypted registered [ 3.466543] ima: No TPM chip found, activating TPM-bypass! [ 3.469515] ima: Allocated hash algorithm: sha1 [ 3.471583] ima: No architecture policies found [ 3.475585] evm: Initialising EVM extended attributes: [ 3.477470] evm: security.selinux [ 3.478507] evm: security.ima [ 3.479480] evm: security.capability [ 3.480586] evm: HMAC attrs: 0x1 [ 3.484239] rtc_cmos 00:05: setting system clock to 2026-02-18 18:29:02 UTC (1771439342) [ 3.489852] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 3.491820] debug: unmapping init [mem 0xffffffffa6582000-0xffffffffa6858fff] [ 3.501283] Write protecting the kernel read-only data: 28672k [ 3.503560] debug: unmapping init [mem 0xffffffffa4c03000-0xffffffffa4dfffff] [ 3.505266] debug: unmapping init [mem 0xffffffffa5514000-0xffffffffa55fffff] [ 3.548187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.554178] systemd[1]: Detected virtualization kvm. [ 3.556250] systemd[1]: Detected architecture x86-64. [ 3.563836] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.600255] systemd[1]: No hostname configured. [ 3.602330] systemd[1]: Set hostname to . [ 3.604716] random: systemd: uninitialized urandom read (16 bytes read) [ 3.607544] systemd[1]: Initializing machine ID from random generator. [ 3.804356] random: systemd: uninitialized urandom read (16 bytes read) [ 3.807269] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.812844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.820115] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.841800] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.659521] device-mapper: uevent: version 1.0.3 [ 4.661728] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 6.073199] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.832919] scsi host0: ata_piix [ 6.842275] scsi host1: ata_piix [ 6.844294] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.854742] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.855939] random: fast init done [ 10.991310] random: crng init done [ 10.992517] random: 7 urandom warning(s) missed due to ratelimiting [ 13.346644] dracut-initqueue[574]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.510049] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.115211] printk: systemd: 25 output lines suppressed due to ratelimiting [ 16.483676] SELinux: Disabled at runtime. [ 16.567989] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.579939] systemd[1]: Detected virtualization kvm. [ 16.582224] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.427093] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.437253] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.459597] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.462926] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.473565] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.484444] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.504137] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 17.746419] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.941564] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.799374] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.818579] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.489715] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.533529] EDAC sbridge: Ver: 1.1.2 [ 22.873156] Key type dns_resolver registered [ 23.311639] NFS: Registering the id_resolver key type [ 23.313688] Key type id_resolver registered [ 23.315017] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg312-client login: [ 53.715395] libcfs: loading out-of-tree module taints kernel. [ 53.772707] Key type ._llcrypt registered [ 53.774081] Key type .llcrypt registered [ 54.011651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 54.017213] alg: No test for adler32 (adler32-zlib) [ 55.002861] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 55.340216] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 56.952193] Key type lgssc registered [ 57.468490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.084974] Lustre: Mounted lustre-client [ 107.465404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.052497] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing check_logdir /tmp/testlogs/ [ 119.530562] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing yml_node [ 120.957785] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 121.842247] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 122.703353] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 123.230295] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Feb 18 13:31:01 EST 2026 [ 128.160407] LustreError: 7880:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d3bc32d5800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 128.174068] Lustre: Unmounted lustre-client [ 135.480660] Key type lgssc unregistered [ 135.596313] LNet: 8325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 135.598919] LNetError: 8325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 135.606191] LNet: Removed LNI 192.168.203.12@tcp [ 135.915128] Key type .llcrypt unregistered [ 135.918210] Key type ._llcrypt unregistered [ 145.528385] Key type ._llcrypt registered [ 145.529945] Key type .llcrypt registered [ 145.594109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 145.600818] alg: No test for adler32 (adler32-zlib) [ 146.401339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 146.428291] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 146.438424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 148.025936] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 148.060063] LNet: 9195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 148.064614] LNetError: 9195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 148.073661] LNet: Removed LNI 192.168.203.12@tcp [ 148.335366] Key type .llcrypt unregistered [ 148.336447] Key type ._llcrypt unregistered [ 150.479183] Key type ._llcrypt registered [ 150.480589] Key type .llcrypt registered [ 150.962337] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:31:29 (1771439489) [ 151.091702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 151.097736] alg: No test for adler32 (adler32-zlib) [ 151.898439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.930137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.937900] LNet: 9611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 151.947809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.968573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 151.976309] LNet: 9618:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 154.398369] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:31:32 (1771439492) [ 154.428534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 154.441027] LNet: 10191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 154.451049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.483185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 156.805803] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:31:35 (1771439495) [ 156.837401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 156.847825] LNet: 10772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 156.858436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 156.896018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 159.304198] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:31:37 (1771439497) [ 159.346533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.356887] LNet: 11359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 159.366817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.400913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 161.744290] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:31:40 (1771439500) [ 161.790940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.812587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.840471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 162.006714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 164.642420] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:31:43 (1771439503) [ 164.681865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 164.691354] LNet: 12533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 164.693908] LNet: 12533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 164.705649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.745442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 164.908529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 165.095241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 167.531668] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:31:46 (1771439506) [ 167.565595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.587491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.630332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 169.865596] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:31:48 (1771439508) [ 169.892615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 169.907870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.937459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 169.946497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 170.083354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 170.096957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 170.227207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 170.240173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 170.387553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 170.402340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 170.531979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 170.544391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 170.695940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 170.710526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 170.857104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 170.872150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 171.028420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 171.042683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 171.187347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 171.198871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 171.330721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 171.340353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 171.482816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 171.499143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 171.642927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 171.659093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 171.783332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 171.798163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 173.941692] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:31:52 (1771439512) [ 173.977343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.988630] LNet: 14389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.991330] LNet: 14389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 174.003970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.037032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 174.053057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 176.196514] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:31:54 (1771439514) [ 176.246924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.265591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.293718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 176.309302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 178.511441] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:31:57 (1771439517) [ 178.548170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.565604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.601652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 178.615634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 181.037726] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:31:59 (1771439519) [ 181.099079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.113704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.145410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 181.160872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 183.400105] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:32:01 (1771439521) [ 183.444575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.460586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.488703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 183.504147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 185.871903] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:32:04 (1771439524) [ 185.912560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 185.937291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.975988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 185.992398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 188.315786] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:32:06 (1771439526) [ 188.357127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.373260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.405711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 188.563575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 190.797839] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:32:09 (1771439529) [ 190.833956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.844045] LNet: 18498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 190.847058] LNet: 18498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 190.858491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.945665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 191.026094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 193.325039] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:32:11 (1771439531) [ 193.405695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.420279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.473515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 193.554116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 196.112236] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:32:14 (1771439534) [ 196.160618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.197645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.369490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 196.481889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 198.901859] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:32:17 (1771439537) [ 198.974656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.989987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.024289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 199.048592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 201.522401] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:32:20 (1771439540) [ 201.571565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.588906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.616133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 201.637917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 204.134148] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:32:22 (1771439542) [ 204.219563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.234386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.261860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 206.631167] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:32:25 (1771439545) [ 206.722097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.736978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.766217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 209.266210] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:32:27 (1771439547) [ 209.322776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.346458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.376136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 209.524952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 212.025123] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:32:30 (1771439550) [ 212.077867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.097378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 212.134334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 212.152858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 214.730595] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:32:33 (1771439553) [ 214.790205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.809860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.844995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 214.864371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 217.193573] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:32:35 (1771439555) [ 217.231420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.259708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.298749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 217.319522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 219.752857] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:32:38 (1771439558) [ 219.790190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.807553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.845666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 219.861143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 222.339991] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:32:40 (1771439560) [ 222.401789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.418664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.439523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 222.461762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 222.469902] LNetError: 25697:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 222.485114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 222.498922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 224.609654] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:32:43 (1771439563) [ 224.645818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.660260] LNet: 26268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 224.662898] LNet: 26268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 224.673362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.712261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E1Yn8Th2Q4 [ 224.742490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E1Yn8Th2Q4 [ 224.763766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E1Yn8Th2Q4 [ 227.083984] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:32:45 (1771439565) [ 227.111660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.129868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.157082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 227.171331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 227.190831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 229.453627] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:32:47 (1771439567) [ 229.491482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.510222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.542098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 230.546842] LNet: Message forwarding will be enabled in 2s [ 233.632135] LNet: Message forwarding enabled [ 233.700186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 233.710340] LNet: Message forwarding will stop in 2s to 6s [ 237.878371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 238.407521] LNet: Message forwarding disabled [ 238.904972] LNet: Message forwarding will be enabled in 2s [ 241.193064] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:32:59 (1771439579) [ 241.224163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.820360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.851524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 244.125674] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:33:02 (1771439582) [ 244.167354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.186620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.213707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 244.228207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 244.242774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 244.258045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 244.271206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 244.287187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 244.304326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 244.322519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 244.340608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 244.359860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 244.377513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 244.390451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 244.404617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 244.419325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 244.434331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 244.449186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 244.464528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 244.477638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 244.490366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 244.499104] LNetError: 28763:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 246.775884] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:33:05 (1771439585) [ 246.814177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.842424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.876921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 249.320618] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:33:07 (1771439587) [ 250.062501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg312-server [ 250.745617] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:33:09 (1771439589) [ 250.798039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.816942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.842167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 250.854496] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 251.567594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[13-17]@tcp [ 251.576359] LNet: 30477:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 254.311851] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:33:12 (1771439592) [ 254.688280] LNetError: 30273:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.15@tcp has gone from up to down [ 254.710801] LNetError: 30267:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 255.198107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg312-server [ 255.721465] LNetError: 30267:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 255.725893] LNetError: 30267:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 255.940221] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:33:14 (1771439594) [ 255.970937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.736232] LNetError: 30266:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.203.17@tcp [ 258.019162] LNetError: 31403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.031591] LNet: Removed LNI 192.168.203.12@tcp [ 258.042080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.064336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 258.075544] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 258.383172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[13-18/2]@tcp [ 258.391918] LNet: 31528:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 258.395103] LNet: 31528:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 258.402054] LNetError: 31415:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 258.403477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[13-18/2]@tcp [ 258.408679] LNetError: 31415:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.17@tcp has gone from up to down [ 258.416323] LNetError: 31415:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 260.902779] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:33:19 (1771439599) [ 260.994105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.006725] LNetError: 32119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.019562] LNet: Removed LNI 192.168.203.12@tcp [ 261.028995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.071637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 261.095319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.108744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.140929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 261.168923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.191851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.241135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 261.288648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.304814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.335838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 261.369846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.385537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.409421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 261.442914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.464417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.502019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 261.537701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.555241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.579865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 261.618761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.634886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.663807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 261.694240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.710471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.731519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 261.763588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.778925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.809993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 261.841704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.855908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.880140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 261.901542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.915579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.941554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 264.389146] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:33:22 (1771439602) [ 265.109152] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg312-server [ 265.804775] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:33:24 (1771439604) [ 266.500242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg312-server [ 267.129049] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:33:25 (1771439605) [ 267.438952] Key type .llcrypt unregistered [ 267.440336] Key type ._llcrypt unregistered [ 269.776415] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 269.851991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 269.855684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 269.859641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 272.041974] Key type ._llcrypt registered [ 272.043190] Key type .llcrypt registered [ 272.107087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 272.113348] alg: No test for adler32 (adler32-zlib) [ 272.930851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.955714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 272.967773] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 273.207958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 273.217599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 274.066953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 274.075356] LNet: Removed LNI 10.1.2.3@tcp [ 274.122520] LNet: 34234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.125644] LNetError: 34234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.135140] LNet: Removed LNI 192.168.203.12@tcp [ 274.392455] Key type .llcrypt unregistered [ 274.393864] Key type ._llcrypt unregistered [ 278.589096] Key type ._llcrypt registered [ 278.590457] Key type .llcrypt registered [ 279.146743] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:33:37 (1771439617) [ 279.333347] Key type .llcrypt unregistered [ 279.335883] Key type ._llcrypt unregistered [ 281.425914] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 281.491933] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 281.496739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 281.499733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 283.736846] Key type ._llcrypt registered [ 283.738305] Key type .llcrypt registered [ 283.824045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.831532] alg: No test for adler32 (adler32-zlib) [ 284.674414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.700204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 284.711398] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 285.045680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 285.053401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 286.267589] LNet: 35726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.272613] LNetError: 35726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.275444] LNet: Removed LNI 192.168.203.12@tcp [ 286.282734] LNet: Removed LNI 10.1.2.3@tcp [ 286.555613] Key type .llcrypt unregistered [ 286.557503] Key type ._llcrypt unregistered [ 291.187304] Key type ._llcrypt registered [ 291.188271] Key type .llcrypt registered [ 291.800259] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:33:50 (1771439630) [ 292.069368] Key type .llcrypt unregistered [ 292.070782] Key type ._llcrypt unregistered [ 294.504231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 294.594931] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 294.597098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 294.599461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 296.806054] Key type ._llcrypt registered [ 296.807211] Key type .llcrypt registered [ 296.865076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 296.871686] alg: No test for adler32 (adler32-zlib) [ 297.675523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.699683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 297.710524] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 298.493117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 298.503084] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 299.306184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 299.314490] LNet: Removed LNI 10.1.2.3@tcp [ 299.321140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 299.339540] LNet: Removed LNI 10.1.2.31@tcp [ 299.347446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 299.358151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 299.630683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 299.642247] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 300.003835] LNet: 37643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.006645] LNetError: 37643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 300.008973] LNet: Removed LNI 10.1.2.3@tcp [ 300.014256] LNet: Removed LNI 10.1.2.31@tcp [ 300.249373] Key type .llcrypt unregistered [ 300.250971] Key type ._llcrypt unregistered [ 304.367333] Key type ._llcrypt registered [ 304.368747] Key type .llcrypt registered [ 304.903680] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:34:03 (1771439643) [ 305.076473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 305.081843] alg: No test for adler32 (adler32-zlib) [ 305.899132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.922179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 305.933613] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 306.275990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.288653] LNet: 38594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.291664] LNetError: 38594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.301468] LNet: Removed LNI 192.168.203.12@tcp [ 306.310511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.331924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.12@tcp [ 306.342079] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 306.637426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.12@tcp [ 306.655396] LNet: Removed LNI 192.168.203.12@tcp [ 306.921320] Key type .llcrypt unregistered [ 306.922665] Key type ._llcrypt unregistered [ 311.146584] Key type ._llcrypt registered [ 311.147719] Key type .llcrypt registered [ 311.658113] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:34:10 (1771439650) [ 312.404856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg312-server [ 313.177659] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:34:11 (1771439651) [ 313.428991] Key type .llcrypt unregistered [ 313.430327] Key type ._llcrypt unregistered [ 315.873793] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 315.955361] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 315.957390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 315.959119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 318.188752] Key type ._llcrypt registered [ 318.190418] Key type .llcrypt registered [ 318.280534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 318.288896] alg: No test for adler32 (adler32-zlib) [ 319.114523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.140809] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 319.453844] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 320.355637] LNet: 40565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.359197] LNetError: 40565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 320.362834] LNet: Removed LNI 192.168.203.12@tcp1 [ 320.370675] LNet: Removed LNI 10.1.2.3@tcp2 [ 320.630687] Key type .llcrypt unregistered [ 320.633122] Key type ._llcrypt unregistered [ 324.751674] Key type ._llcrypt registered [ 324.753170] Key type .llcrypt registered [ 325.252508] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:34:23 (1771439663) [ 325.419236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 325.426293] alg: No test for adler32 (adler32-zlib) [ 326.256479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.286080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 326.299620] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 326.691976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.700971] LNet: 41532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.704838] LNetError: 41532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.716392] LNet: Removed LNI 192.168.203.12@tcp [ 326.724185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.750839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 326.766567] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 329.415383] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:34:27 (1771439667) [ 329.456427] LNet: 42115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.458820] LNetError: 42115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.469878] LNet: Removed LNI 192.168.203.12@tcp [ 329.762498] Key type .llcrypt unregistered [ 329.763622] Key type ._llcrypt unregistered [ 331.875994] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 331.916124] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 331.918255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 331.920434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 334.133407] Key type ._llcrypt registered [ 334.134514] Key type .llcrypt registered [ 334.204219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 334.212412] alg: No test for adler32 (adler32-zlib) [ 335.022324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.050743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 335.062117] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 335.366869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 335.374893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 336.235295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.244804] LNet: 42857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.248797] LNetError: 42857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.251672] LNet: Removed LNI 192.168.203.12@tcp [ 336.257384] LNet: Removed LNI 10.1.2.3@tcp [ 336.270483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.298057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 336.309697] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 336.311898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 336.507645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.519266] LNet: Removed LNI 192.168.203.12@tcp [ 336.528385] LNet: Removed LNI 10.1.2.3@tcp [ 336.537086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.563417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 336.574152] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 336.576692] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 336.771054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 336.779700] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 337.188823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.197786] LNet: 43048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.200569] LNet: 43048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 337.203790] LNetError: 43048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 337.206794] LNetError: 43048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 337.211717] LNet: Removed LNI 192.168.203.12@tcp [ 337.213865] LNet: Removed LNI 10.1.2.3@tcp [ 337.226289] LNet: Removed LNI 192.168.203.12@tcp2 [ 337.233847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.257321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 337.267647] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 337.269961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 337.272819] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 340.049212] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:34:38 (1771439678) [ 340.112663] LNet: 43639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.115841] LNetError: 43639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.119071] LNetError: 43639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 340.123781] LNet: Removed LNI 192.168.203.12@tcp [ 340.125527] LNet: Removed LNI 10.1.2.3@tcp [ 340.135613] LNet: Removed LNI 192.168.203.12@tcp2 [ 340.407671] Key type .llcrypt unregistered [ 340.408891] Key type ._llcrypt unregistered [ 342.746354] Key type ._llcrypt registered [ 342.747568] Key type .llcrypt registered [ 342.859953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.867225] alg: No test for adler32 (adler32-zlib) [ 343.703364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.737034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 343.753476] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 344.116354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.125395] LNet: 44041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.128175] LNetError: 44041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.141335] LNet: Removed LNI 192.168.203.12@tcp [ 344.154655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.183331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 344.196355] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 344.376638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.389689] LNet: Removed LNI 192.168.203.12@tcp [ 344.402398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.429641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 344.440462] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 344.610411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.628293] LNet: Removed LNI 192.168.203.12@tcp [ 344.635522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.654526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 344.664809] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 344.821749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 344.857321] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 344.859663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 344.861996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 347.019761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.030101] LNet: 44173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.032206] LNet: 44173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 347.034382] LNetError: 44173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.037076] LNetError: 44173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 347.046393] LNet: Removed LNI 192.168.203.12@tcp [ 347.055923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.089602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 347.105598] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 349.667232] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:34:48 (1771439688) [ 349.725594] LNet: 44767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.728883] LNetError: 44767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 349.741060] LNet: Removed LNI 192.168.203.12@tcp [ 349.973665] Key type .llcrypt unregistered [ 349.975098] Key type ._llcrypt unregistered [ 352.075700] Key type ._llcrypt registered [ 352.077022] Key type .llcrypt registered [ 352.143904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 352.151341] alg: No test for adler32 (adler32-zlib) [ 352.979210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.003910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 353.014454] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 353.396478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.406710] LNet: 45167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.409882] LNetError: 45167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 353.418319] LNet: Removed LNI 192.168.203.12@tcp [ 353.426804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.454987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 353.474406] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 356.046806] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:34:54 (1771439694) [ 356.083519] LNet: 45751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.087816] LNetError: 45751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.097115] LNet: Removed LNI 192.168.203.12@tcp [ 356.349585] Key type .llcrypt unregistered [ 356.351490] Key type ._llcrypt unregistered [ 358.314034] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 358.385429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 358.391744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 358.394240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 360.572988] Key type ._llcrypt registered [ 360.574636] Key type .llcrypt registered [ 360.640318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 360.646977] alg: No test for adler32 (adler32-zlib) [ 361.452175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.474358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 361.485098] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 361.813305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.820865] LNet: 46209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.823064] LNetError: 46209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 361.832527] LNet: Removed LNI 192.168.203.12@tcp [ 361.842748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.868791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 361.881302] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 364.428700] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:35:02 (1771439702) [ 364.505900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.513808] LNet: 46806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.517685] LNetError: 46806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.531401] LNet: Removed LNI 192.168.203.12@tcp [ 364.540442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.590579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 366.981147] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:35:05 (1771439705) [ 367.038979] LNet: 47386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.286514] Key type .llcrypt unregistered [ 367.288696] Key type ._llcrypt unregistered [ 369.350415] Key type ._llcrypt registered [ 369.351544] Key type .llcrypt registered [ 369.429969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.436789] alg: No test for adler32 (adler32-zlib) [ 370.244600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.266736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 370.282972] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 370.618087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.626913] LNet: 47785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.632026] LNetError: 47785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.640825] LNet: Removed LNI 192.168.203.12@tcp [ 370.649128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.677323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 370.689821] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 373.433733] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:35:11 (1771439711) [ 373.521840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.531475] LNet: 48378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.534406] LNetError: 48378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 373.548055] LNet: Removed LNI 192.168.203.12@tcp [ 373.557956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.586929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 375.959877] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:35:14 (1771439714) [ 376.642585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 377.264039] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:35:15 (1771439715) [ 377.911376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 378.500401] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:35:17 (1771439717) [ 379.105428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 379.772492] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:35:18 (1771439718) [ 380.410293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 381.057460] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:35:19 (1771439719) [ 381.614796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 382.226512] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:35:20 (1771439720) [ 382.261325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.272232] LNet: 50421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.284392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.312683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 382.321345] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 382.608515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.12 [ 382.616273] LNet: 50560:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 382.621144] LNet: Removed LNI 192.168.203.12@tcp [ 384.972110] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:35:23 (1771439723) [ 385.014566] LNet: 51126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.017258] LNetError: 51126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.023851] LNet: Removed LNI 192.168.203.12@tcp [ 385.275905] Key type .llcrypt unregistered [ 385.277680] Key type ._llcrypt unregistered [ 387.291902] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 387.409612] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 387.411789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 387.414034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 389.708811] Key type ._llcrypt registered [ 389.710011] Key type .llcrypt registered [ 389.771661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.778305] alg: No test for adler32 (adler32-zlib) [ 390.741033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 392.995095] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:35:31 (1771439731) [ 393.028891] LNet: 52028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.033241] LNetError: 52028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 393.043382] LNet: Removed LNI 10.1.2.3@tcp [ 393.274434] Key type .llcrypt unregistered [ 393.275724] Key type ._llcrypt unregistered [ 395.307162] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 395.425879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 395.428309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 395.431038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 397.780142] Key type ._llcrypt registered [ 397.781490] Key type .llcrypt registered [ 397.883404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 397.895636] alg: No test for adler32 (adler32-zlib) [ 398.736673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 401.258919] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:35:39 (1771439739) [ 401.300224] LNet: 52919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.305385] LNetError: 52919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 401.317466] LNet: Removed LNI 10.1.2.3@tcp [ 401.586384] Key type .llcrypt unregistered [ 401.589628] Key type ._llcrypt unregistered [ 403.839234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 403.943555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 403.945580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 403.947743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 406.230904] Key type ._llcrypt registered [ 406.232363] Key type .llcrypt registered [ 406.290629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.297682] alg: No test for adler32 (adler32-zlib) [ 407.121789] LNet: 53226:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 407.127707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 409.457561] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:35:47 (1771439747) [ 409.516256] LNet: 53815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.519805] LNetError: 53815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 409.528275] LNet: Removed LNI 10.1.2.3@tcp [ 409.757513] Key type .llcrypt unregistered [ 409.758971] Key type ._llcrypt unregistered [ 411.690733] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 411.805725] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 411.807659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 411.810114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 414.103498] Key type ._llcrypt registered [ 414.104544] Key type .llcrypt registered [ 414.168167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 414.174871] alg: No test for adler32 (adler32-zlib) [ 414.984814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.022806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 418.196426] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:35:56 (1771439756) [ 418.342627] LNet: 55045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.345959] LNetError: 55045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 418.356480] LNet: Removed LNI 10.1.2.3@tcp [ 418.584337] Key type .llcrypt unregistered [ 418.586161] Key type ._llcrypt unregistered [ 420.730685] Key type ._llcrypt registered [ 420.732096] Key type .llcrypt registered [ 420.818083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 420.824692] alg: No test for adler32 (adler32-zlib) [ 421.646603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 421.662072] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 424.727140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 424.978953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 425.002343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 425.020763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 425.035034] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 425.121752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 425.136645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 425.151849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 425.166702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 425.182213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 425.197491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 425.213262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 425.228289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 425.236417] LNet: 55481:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 425.244126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 425.258017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 425.275747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 425.289703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 425.304569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.324832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.338102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 425.526808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 425.540727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 425.554287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 425.563733] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 425.647444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 425.662919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 425.678524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 425.690058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 425.702212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 425.717956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 425.730481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 425.743589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 425.752050] LNet: 55590:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 425.760196] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 425.760302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 425.781122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 425.792951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 425.806656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 425.819752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.834602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.848227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 426.033362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 426.048657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 426.064329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 426.163813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 426.175782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 426.189772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 426.203443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 426.218177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 426.232399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 426.244134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 426.259686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 426.274840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 426.291940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 426.310253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 426.325544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 426.343237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 426.361302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 426.373564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 426.562830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 426.577559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 426.594696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 426.603393] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 426.606064] LNet: Skipped 1 previous similar message [ 426.688626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 426.703130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 426.716279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 426.728733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 426.741773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 426.754239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 426.770177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 426.786285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 426.794713] LNet: 55808:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 426.797687] LNet: 55808:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 426.805860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 426.819517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 426.834925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 426.848534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 426.862768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 426.876755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 426.890053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 427.070730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 427.087919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 427.104584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 427.193157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 427.206485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 427.220327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 427.234468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 427.248537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 427.262898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 427.279965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 427.296134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 427.310586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 427.322666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 427.333919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 427.345058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 427.358843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.371910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 427.385862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 427.567294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 427.583015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 427.596993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 427.679314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 427.691948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 427.703780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 427.715954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 427.728967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 427.742429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 427.756327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 427.769796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 427.781419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 427.793689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 427.805612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 427.817453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 427.829377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.841503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 427.855260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 427.946238] LNet: 56067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.948856] LNetError: 56067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.966353] LNet: Removed LNI 192.168.203.12@tcp [ 428.104599] Key type .llcrypt unregistered [ 428.105823] Key type ._llcrypt unregistered [ 433.358018] Key type ._llcrypt registered [ 433.359106] Key type .llcrypt registered [ 433.838808] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:36:12 (1771439772) [ 434.147272] Key type .llcrypt unregistered [ 434.148969] Key type ._llcrypt unregistered [ 436.226275] Key type ._llcrypt registered [ 436.228231] Key type .llcrypt registered [ 436.305800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 436.312536] alg: No test for adler32 (adler32-zlib) [ 437.119190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 437.134968] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 440.070882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 443.607751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.623260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 443.632667] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 444.122638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 444.136910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 1 [ 444.152858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 444.166791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 1 [ 444.180610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_interrupt -r 1 [ 444.193857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 1 [ 444.208207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_interrupt -r 1 [ 444.224699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 1 [ 444.241219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 444.250432] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 444.254349] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 444.331383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 444.344256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 444.356145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 444.370487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 444.384723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 444.400166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 444.412287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 444.425210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 444.433149] LNet: 57529:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 444.441696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 444.452981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 444.465373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 444.477015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 444.489119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 444.502399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 444.517162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 444.689987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 444.704712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 1 [ 444.720385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 444.735641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 1 [ 444.751192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_dropped -r 1 [ 444.768695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 1 [ 444.785693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_dropped -r 1 [ 444.802444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 1 [ 444.817469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 444.827348] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 444.830433] LNet: Skipped 2 previous similar messages [ 444.915130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 444.927847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 444.939767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 444.951176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 444.963835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 444.978872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 444.992770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 445.007140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 445.014765] LNet: 57650:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 445.023585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 445.037594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 445.051666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 445.065601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 445.079387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 445.095459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 445.117837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 445.327700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 445.343963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 1 [ 445.360546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 445.374392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 1 [ 445.389683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_aborted -r 1 [ 445.404215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 1 [ 445.420183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_aborted -r 1 [ 445.435716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 1 [ 445.451350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 445.546698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 445.563409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 445.582619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 445.605252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 445.619248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 445.632779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 445.648707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 445.668708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 445.682790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 445.700659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 445.714639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 445.730182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 445.745325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 445.760202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 445.775291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 445.967195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 445.981165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 1 [ 445.995803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 446.010103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 1 [ 446.024995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_no_route -r 1 [ 446.040413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 1 [ 446.055655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_no_route -r 1 [ 446.069422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 1 [ 446.084901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 446.093238] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 446.095571] LNet: Skipped 6 previous similar messages [ 446.184950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 446.197898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 446.211862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 446.231730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 446.246612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 446.260972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 446.281605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 446.298504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 446.305319] LNet: 57892:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 446.308393] LNet: 57892:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 446.318527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 446.337628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 446.349746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 446.362990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 446.379823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 446.395578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 446.414225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 446.632200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 446.653453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 1 [ 446.671165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 446.695369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 1 [ 446.712975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_timeout -r 1 [ 446.737151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 1 [ 446.756390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_timeout -r 1 [ 446.775137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 1 [ 446.797719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 446.892714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 446.907151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 446.921195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 446.934834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 446.949692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 446.966473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 446.983613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 446.998547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 447.011767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 447.025267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 447.040564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 447.055408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 447.068822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.088883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.105683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 447.326074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 447.342331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_error -r 1 [ 447.359455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 447.378175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_error -r 1 [ 447.395598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_error -r 1 [ 447.416690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_error -r 1 [ 447.432100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_error -r 1 [ 447.447634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_error -r 1 [ 447.463072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 447.552113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 447.566267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 447.579882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 447.592514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 447.605969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 447.619812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 447.632636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 447.647359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 447.660336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 447.673392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 447.689285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 447.707582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 447.720168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.734776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.750732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 449.983341] LNet: 58210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.986948] LNetError: 58210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.990758] LNet: Removed LNI 192.168.203.12@tcp [ 451.047017] LNet: Removed LNI 192.168.203.12@tcp1 [ 451.212591] Key type .llcrypt unregistered [ 451.213724] Key type ._llcrypt unregistered [ 456.966962] Key type ._llcrypt registered [ 456.968185] Key type .llcrypt registered [ 457.441455] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:36:35 (1771439795) [ 457.737252] Key type .llcrypt unregistered [ 457.738597] Key type ._llcrypt unregistered [ 459.761635] Key type ._llcrypt registered [ 459.763342] Key type .llcrypt registered [ 459.843319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 459.850776] alg: No test for adler32 (adler32-zlib) [ 460.658500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 460.672311] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 463.638688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 463.885485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 463.902354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 463.920234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 463.930133] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 464.011262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 464.025710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 464.039531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 464.053564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 464.069377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 464.084678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 464.099754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 464.113505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 464.122581] LNet: 59455:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 464.132038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 464.147479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 464.161951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 464.177634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 464.192936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.206770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 464.221610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 464.405471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 464.420378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 464.434488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 464.441392] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 464.517269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 464.529587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 464.541200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 464.554926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 464.566809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 464.578578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 464.589181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 464.601634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 464.612422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 464.623179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 464.634428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 464.645804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 464.659495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.673453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 464.684839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 464.873021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 464.886877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 464.901372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 464.986952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 464.998805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 465.012334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 465.025111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 465.038106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 465.052053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 465.064221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 465.077137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 465.083355] LNet: 59673:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 465.085532] LNet: 59673:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 465.091135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 465.102393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 465.112982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 465.123334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 465.136400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 465.149830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 465.166679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 465.269524] LNet: 59714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.272956] LNetError: 59714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.288312] LNet: Removed LNI 192.168.203.12@tcp [ 465.452392] Key type .llcrypt unregistered [ 465.453631] Key type ._llcrypt unregistered [ 470.937080] Key type ._llcrypt registered [ 470.938238] Key type .llcrypt registered [ 471.397943] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:36:49 (1771439809) [ 471.696470] Key type .llcrypt unregistered [ 471.697445] Key type ._llcrypt unregistered [ 473.688120] Key type ._llcrypt registered [ 473.689723] Key type .llcrypt registered [ 473.753589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 473.761343] alg: No test for adler32 (adler32-zlib) [ 474.566462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 474.577637] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 477.597669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 480.463752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.482955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 480.493517] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 481.026791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 481.042101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 1 [ 481.056578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 481.072904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 1 [ 481.092398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_dropped -r 1 [ 481.109273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 1 [ 481.124942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_dropped -r 1 [ 481.141970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 1 [ 481.160783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 481.170088] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 481.175035] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 481.269432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 481.290681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 481.309904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 481.324545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 481.340815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 481.354886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 481.370791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 481.384663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 481.394742] LNet: 61176:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 481.404376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 481.419970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 481.434122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 481.447025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 481.460877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 481.477850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 481.491675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 481.506231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 481.710830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 481.726318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_error -r 1 [ 481.745823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 481.768712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_error -r 1 [ 481.785866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_error -r 1 [ 481.804600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_error -r 1 [ 481.823512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_error -r 1 [ 481.840960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_error -r 1 [ 481.858336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 481.867701] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 481.870792] LNet: Skipped 2 previous similar messages [ 481.946826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 481.959142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 481.971425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 481.982721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 481.995728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 482.009502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 482.024907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 482.037792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 482.046694] LNet: 61299:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 482.054579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 482.067218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 482.080557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 482.093516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 482.107332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.120994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 482.137255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 482.155879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.366144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 482.382230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_timeout -r 1 [ 482.398398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 482.413452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_timeout -r 1 [ 482.428331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_timeout -r 1 [ 482.446991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_timeout -r 1 [ 482.461480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_timeout -r 1 [ 482.477645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_timeout -r 1 [ 482.495039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.112@tcp [ 482.590731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 482.607653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 482.622201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 482.635939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 482.656361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 482.669750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 482.681823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 482.694750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 482.708159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 482.721689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 482.736867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 482.751641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 482.768684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.782273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 482.795684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 482.810110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 485.101795] LNet: 61500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.104811] LNetError: 61500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 485.108742] LNet: Removed LNI 192.168.203.12@tcp [ 485.116320] LNet: Removed LNI 192.168.203.12@tcp1 [ 485.265344] Key type .llcrypt unregistered [ 485.266514] Key type ._llcrypt unregistered [ 490.864073] Key type ._llcrypt registered [ 490.865555] Key type .llcrypt registered [ 491.354971] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:37:09 (1771439829) [ 491.533322] Key type .llcrypt unregistered [ 491.534297] Key type ._llcrypt unregistered [ 493.601037] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 493.669256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 493.670936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 493.672826] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 495.877020] Key type ._llcrypt registered [ 495.878527] Key type .llcrypt registered [ 495.944417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.950182] alg: No test for adler32 (adler32-zlib) [ 496.750700] LNet: 62574:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 496.755089] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 496.766331] LNet: 62588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.769046] LNetError: 62588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.776341] LNet: Removed LNI 192.168.203.12@tcp [ 497.026489] Key type .llcrypt unregistered [ 497.028443] Key type ._llcrypt unregistered [ 499.133839] Key type ._llcrypt registered [ 499.135552] Key type .llcrypt registered [ 499.221199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 499.230160] alg: No test for adler32 (adler32-zlib) [ 500.053072] LNet: 62846:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 500.058363] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 500.059890] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 500.074276] LNet: 62860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.077107] LNetError: 62860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.079471] LNet: Removed LNI 192.168.203.12@tcp [ 500.090349] LNet: Removed LNI 10.1.2.3@tcp1 [ 500.376346] Key type .llcrypt unregistered [ 500.377767] Key type ._llcrypt unregistered [ 502.567103] Key type ._llcrypt registered [ 502.568306] Key type .llcrypt registered [ 502.630875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 502.636440] alg: No test for adler32 (adler32-zlib) [ 503.454666] LNet: 63119:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 503.462579] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 503.464497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 503.483695] LNet: 63133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.487206] LNetError: 63133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.493241] LNet: Removed LNI 192.168.203.12@tcp [ 503.501436] LNet: Removed LNI 10.1.2.3@tcp [ 503.787719] Key type .llcrypt unregistered [ 503.789601] Key type ._llcrypt unregistered [ 505.857972] Key type ._llcrypt registered [ 505.859367] Key type .llcrypt registered [ 505.917560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 505.923218] alg: No test for adler32 (adler32-zlib) [ 506.728302] LNet: 63392:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 506.733830] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 506.735582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 506.750025] LNet: 63406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.753167] LNetError: 63406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 506.755461] LNet: Removed LNI 192.168.203.12@tcp [ 506.761514] LNet: Removed LNI 10.1.2.3@tcp [ 507.022267] Key type .llcrypt unregistered [ 507.023622] Key type ._llcrypt unregistered [ 509.217332] Key type ._llcrypt registered [ 509.218791] Key type .llcrypt registered [ 509.288944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 509.294696] alg: No test for adler32 (adler32-zlib) [ 510.110179] LNet: 63670:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 510.116256] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 510.118435] LNetError: 63670:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 510.120957] LNetError: Error -100 starting up LNI tcp [ 510.128355] LNet: Removed LNI 192.168.203.12@tcp [ 510.385395] Key type .llcrypt unregistered [ 510.386625] Key type ._llcrypt unregistered [ 514.324954] Key type ._llcrypt registered [ 514.326500] Key type .llcrypt registered [ 514.811407] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:37:33 (1771439853) [ 515.147709] Key type .llcrypt unregistered [ 515.148853] Key type ._llcrypt unregistered [ 517.188453] Key type ._llcrypt registered [ 517.189809] Key type .llcrypt registered [ 517.276580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.285581] alg: No test for adler32 (adler32-zlib) [ 518.089177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 518.101033] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 520.951088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 521.177492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 521.193110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 521.207403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 521.215011] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 521.293063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 521.304459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 521.316730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 521.329737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 521.342430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 521.354835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 521.366516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 521.378287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 521.384346] LNet: 64922:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 521.391570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 521.404166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 521.417302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 521.430324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 521.441697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 521.455788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 521.550289] LNet: 64961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.553427] LNetError: 64961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.562956] LNet: Removed LNI 192.168.203.12@tcp [ 521.707411] Key type .llcrypt unregistered [ 521.708712] Key type ._llcrypt unregistered [ 527.288099] Key type ._llcrypt registered [ 527.289671] Key type .llcrypt registered [ 527.364484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 527.371696] alg: No test for adler32 (adler32-zlib) [ 528.174323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 528.186035] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 531.196900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 534.140739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.156896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 534.166919] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 534.679195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 534.694357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e network_timeout -r 1 [ 534.709852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 534.724154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e network_timeout -r 1 [ 534.746159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e network_timeout -r 1 [ 534.759710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e network_timeout -r 1 [ 534.775194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e network_timeout -r 1 [ 534.797137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e network_timeout -r 1 [ 534.815764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 534.825506] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 534.912350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 534.928560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 534.943568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 534.957858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 534.971584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 534.983692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 534.998941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 535.013606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 535.021599] LNet: 65837:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 535.029656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 535.045807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 535.061353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 535.077307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 535.090843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.107209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 537.225972] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 537.331640] LNet: 65910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.334930] LNetError: 65910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.337984] LNet: Removed LNI 192.168.203.12@tcp [ 537.345335] LNet: Removed LNI 192.168.203.12@tcp1 [ 537.504590] Key type .llcrypt unregistered [ 537.506270] Key type ._llcrypt unregistered [ 543.084395] Key type ._llcrypt registered [ 543.085354] Key type .llcrypt registered [ 543.623036] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:38:02 (1771439882) [ 543.769846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.778504] alg: No test for adler32 (adler32-zlib) [ 544.591797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.616742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 544.627491] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 544.902729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 544.910376] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 545.227771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 545.255697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 545.277503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 545.295750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 545.313316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 545.330049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 545.344859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 545.354846] LNetError: 67007:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 545.632303] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 545.632339] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 547.680363] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 548.515981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 551.776328] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 552.718717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 552.731592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 552.753789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.761550] LNet: 67124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.763906] LNetError: 67124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.765847] LNet: Removed LNI 192.168.203.12@tcp [ 552.776316] LNet: Removed LNI 192.168.203.12@tcp1 [ 552.783538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.802221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 552.810609] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 553.053642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 553.060787] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 553.365778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 553.382964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 553.408532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 553.426838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 553.443491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 553.457590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 553.472756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 553.487863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 553.496739] LNetError: 67396:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 553.824291] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 556.654524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 559.968370] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 559.973813] LNet: Skipped 1 previous similar message [ 563.948801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 563.963073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 563.977446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 566.294879] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:38:24 (1771439904) [ 566.347955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.356850] LNet: 68112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.359396] LNetError: 68112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.362070] LNetError: 68112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 566.364280] LNet: Removed LNI 192.168.203.12@tcp [ 566.369534] LNet: Removed LNI 192.168.203.12@tcp1 [ 566.377272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.395880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 566.404535] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 566.651314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 566.659134] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 566.975243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 567.001808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 567.015274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 567.029720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 567.043865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 567.057428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 567.072155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 0 [ 567.110818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 567.392089] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.12@tcp [ 568.416295] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 568.418674] LNet: Skipped 1 previous similar message [ 583.566289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 583.611836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 583.624494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 583.635341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 500 [ 583.672315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 590.911910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 590.938940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.948515] LNet: 68627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.950943] LNetError: 68627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.953395] LNetError: 68627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 590.956592] LNet: Removed LNI 192.168.203.12@tcp [ 590.970268] LNet: Removed LNI 192.168.203.12@tcp1 [ 590.978591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.999349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.008844] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 591.276761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 591.284577] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 591.611354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 591.625932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 591.648692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 591.661118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 591.675423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 591.688709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 591.702638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 591.716226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.12@tcp --health 0 [ 592.032293] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 592.035286] LNet: Skipped 3 previous similar messages [ 592.825691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 603.224308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 603.236527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 603.247912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 605.561451] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:39:04 (1771439944) [ 605.706032] LNet: 69683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.708431] LNetError: 69683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.711634] LNetError: 69683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 605.715519] LNet: Removed LNI 192.168.203.12@tcp [ 605.722538] LNet: Removed LNI 192.168.203.12@tcp1 [ 605.959786] Key type .llcrypt unregistered [ 605.961045] Key type ._llcrypt unregistered [ 607.849825] Key type ._llcrypt registered [ 607.850979] Key type .llcrypt registered [ 607.916390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 607.922282] alg: No test for adler32 (adler32-zlib) [ 608.725577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 608.735300] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 612.206394] Lustre: DEBUG MARKER: Initial discovery [ 612.250632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 613.055265] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 613.099920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 613.116838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 613.126652] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 613.134621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 613.151148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.112@tcp -r 1 -m GET -l 3 [ 613.166975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.112@tcp -r 1 -m GET -e local_error [ 613.182035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.112@tcp -r 1 -m PUT -l 6 [ 613.690476] Lustre: DEBUG MARKER: Force oleg312-client.virtnet to discover 192.168.203.112@tcp (in background) [ 613.714362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.112@tcp [ 613.856100] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 614.240934] Lustre: DEBUG MARKER: Force oleg312-server to discover 192.168.203.12@tcp [ 615.053422] Lustre: DEBUG MARKER: Wait for 70434 [ 616.736231] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 617.259692] Lustre: DEBUG MARKER: Finished wait on 70434 [ 617.318074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 617.338224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 617.454441] LNet: 70891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.459426] LNetError: 70891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.472308] LNet: Removed LNI 192.168.203.12@tcp [ 617.623417] Key type .llcrypt unregistered [ 617.624702] Key type ._llcrypt unregistered [ 623.014855] Key type ._llcrypt registered [ 623.016072] Key type .llcrypt registered [ 623.533782] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:39:21 (1771439961) [ 623.713234] Key type .llcrypt unregistered [ 623.714552] Key type ._llcrypt unregistered [ 625.811326] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 625.893196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 625.895025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 625.896915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 628.089295] Key type ._llcrypt registered [ 628.090136] Key type .llcrypt registered [ 628.159142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 628.165228] alg: No test for adler32 (adler32-zlib) [ 628.969503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.996832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.007132] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 629.266574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 629.274991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 632.072071] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:39:30 (1771439970) [ 632.113384] LNet: 72958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.116294] LNetError: 72958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.118779] LNet: Removed LNI 192.168.203.12@tcp [ 632.124674] LNet: Removed LNI 10.1.2.3@tcp [ 632.340295] Key type .llcrypt unregistered [ 632.341241] Key type ._llcrypt unregistered [ 634.466510] Key type ._llcrypt registered [ 634.467603] Key type .llcrypt registered [ 634.540051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.545930] alg: No test for adler32 (adler32-zlib) [ 635.350456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.372746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.383591] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 635.659051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 635.666585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 635.990127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.12@tcp [ 637.024157] LNetError: Unexpected error -22 connecting to 192.168.203.12@tcp at host 192.168.203.12:988 [ 638.183219] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:39:36 (1771439976) [ 638.217129] LNet: 74085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.220498] LNetError: 74085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.223551] LNet: Removed LNI 192.168.203.12@tcp [ 638.232718] LNet: Removed LNI 10.1.2.3@tcp [ 638.463329] Key type .llcrypt unregistered [ 638.464616] Key type ._llcrypt unregistered [ 640.348782] Key type ._llcrypt registered [ 640.350196] Key type .llcrypt registered [ 640.422870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.429285] alg: No test for adler32 (adler32-zlib) [ 641.237414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.259715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 641.269940] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 641.547056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 641.555929] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 641.894772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.12@tcp1 --nid 192.168.203.12@tcp2 [ 643.391595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp1 192.168.203.12@tcp2 [ 643.400797] LNetError: 74925:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.12@tcp1 to 192.168.203.12@tcp2 - Route table may be misconfigured [ 643.412462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp2 192.168.203.12@tcp1 [ 645.647548] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:39:44 (1771439984) [ 645.677880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.688732] LNet: 75493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.692528] LNetError: 75493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.696633] LNet: Removed LNI 192.168.203.12@tcp1 [ 646.758568] LNet: Removed LNI 192.168.203.12@tcp2 [ 646.771932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.794127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.803898] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 647.043087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 647.051830] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 647.329563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 647.344312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.12@tcp [ 647.356340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp1 192.168.203.12@tcp1 [ 647.370772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.12@tcp -d 192.168.203.12@tcp -e network_timeout [ 647.382873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -e network_timeout [ 647.396296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -e network_timeout [ 647.410171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -e network_timeout [ 647.423794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 647.431317] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 647.438904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 647.469835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 647.502253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 647.840087] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 649.743362] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:39:48 (1771439988) [ 649.779885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.790857] LNet: 76355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.793951] LNetError: 76355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.799401] LNetError: 76355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 649.803281] LNet: Removed LNI 192.168.203.12@tcp [ 650.853961] LNet: Removed LNI 192.168.203.12@tcp1 [ 650.865902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.904375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 651.005252] LNet: 76397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.154326] Key type .llcrypt unregistered [ 651.155702] Key type ._llcrypt unregistered [ 654.912552] Key type ._llcrypt registered [ 654.914197] Key type .llcrypt registered [ 655.395461] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:39:53 (1771439993) [ 655.532902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.540204] alg: No test for adler32 (adler32-zlib) [ 656.344908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.856110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 657.509631] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:39:56 (1771439996) [ 657.544807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.555357] LNet: 77488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.565645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.592528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.609566] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 657.908375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.917355] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 658.273553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 658.294570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp1 [ 658.311660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp1 [ 660.631948] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:39:59 (1771439999) [ 661.345396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg312-server [ 661.994514] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:40:00 (1771440000) [ 662.631801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg312-server [ 663.213437] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:40:01 (1771440001) [ 663.831232] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg312-server [ 664.382180] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:40:02 (1771440002) [ 664.992402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg312-server [ 665.623206] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:40:04 (1771440004) [ 666.239564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg312-server [ 666.805771] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:40:05 (1771440005) [ 667.434115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg312-server [ 667.953580] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:40:06 (1771440006) [ 668.571956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg312-server [ 669.117953] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:40:07 (1771440007) [ 669.758697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg312-server [ 670.321374] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:40:08 (1771440008) [ 670.941066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg312-server [ 671.529185] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:40:10 (1771440010) [ 671.599360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.619787] LNet: 81368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.624045] LNetError: 81368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.626772] LNet: Removed LNI 192.168.203.12@tcp [ 672.678506] LNet: Removed LNI 192.168.203.12@tcp1 [ 672.688657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.715163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.725716] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 672.992518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 673.030275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 673.097401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.106468] LNet: 81515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.109333] LNetError: 81515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.111735] LNetError: 81515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 673.123445] LNet: Removed LNI 192.168.203.12@tcp [ 673.131509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.153616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 673.164937] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 673.445174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 673.508759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.536262] LNet: Removed LNI 192.168.203.12@tcp [ 674.545917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.567226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.577553] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 674.865619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 674.896448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 674.957927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.966225] LNetError: 81808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.968664] LNetError: 81808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 674.975228] LNet: Removed LNI 192.168.203.12@tcp [ 674.982728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.000407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 675.012797] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 675.291028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 675.373856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.382498] LNet: 81953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.385111] LNet: 81953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 676.454665] LNet: Removed LNI 192.168.203.12@tcp [ 676.468543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.492728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.503073] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 676.775569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 676.808382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 676.883460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.926310] LNet: Removed LNI 192.168.203.12@tcp [ 677.933969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.954191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 677.964880] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 678.216110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 678.290043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.299931] LNetError: 82245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.302020] LNetError: 82245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 678.312350] LNet: Removed LNI 192.168.203.12@tcp [ 678.320470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.341354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.352390] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 678.605869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 678.636183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 678.733771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.782312] LNet: Removed LNI 192.168.203.12@tcp [ 679.791548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.817190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 679.828633] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 680.104834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 680.189204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.198043] LNet: 82538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.200347] LNet: 82538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 681.254350] LNet: Removed LNI 192.168.203.12@tcp [ 681.262513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.286286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.296097] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 681.551131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 681.585739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 681.669280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.726460] LNet: Removed LNI 192.168.203.12@tcp [ 682.736954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.760579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 682.772194] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 683.045307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 683.120725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.129629] LNetError: 82830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.132143] LNetError: 82830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 683.140512] LNet: Removed LNI 192.168.203.12@tcp [ 683.148396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.170364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.180268] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 683.422265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 683.450532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 683.522017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.586390] LNet: Removed LNI 192.168.203.12@tcp [ 684.594167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.614434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 684.624964] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 684.903283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 685.000951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.054318] LNet: Removed LNI 192.168.203.12@tcp [ 686.062788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.085351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.097479] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 686.392494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 686.419456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.442337] LNet: Removed LNI 192.168.203.12@tcp [ 686.451737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.474939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.484644] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 686.773266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 689.241725] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:40:27 (1771440027) [ 689.270912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.281396] LNet: 83952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.283691] LNet: 83952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 689.292995] LNet: Removed LNI 192.168.203.12@tcp [ 689.303691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.329805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 689.340274] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 689.797231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.810328] LNet: Removed LNI 192.168.203.12@tcp231 [ 689.818976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.847389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 689.860073] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 689.862386] LNet: 84104:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 690.006828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 690.026435] LNet: Removed LNI 192.168.203.12@tcp231 [ 690.033523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 690.045084] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 690.460472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.475464] LNet: Removed LNI 192.168.203.12@tcp231 [ 690.483405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.517911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 690.527757] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/0] [ 690.530317] LNet: 84256:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 690.673702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 690.691505] LNet: Removed LNI 192.168.203.12@tcp231 [ 690.698122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 690.708874] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/0] [ 691.125116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.132688] LNetError: 84399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.135497] LNetError: 84399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 691.152097] LNet: Removed LNI 192.168.203.12@tcp231 [ 691.159933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.187029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 691.197439] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/60] [ 691.353532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 691.371490] LNet: Removed LNI 192.168.203.12@tcp231 [ 691.377965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 691.388604] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/60] [ 691.837144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.850499] LNet: Removed LNI 192.168.203.12@tcp231 [ 691.859179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.883217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 691.893197] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 691.895826] LNet: 84558:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 691.898611] LNet: 84558:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 692.039308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 692.057521] LNet: Removed LNI 192.168.203.12@tcp231 [ 692.064247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 692.074687] LNet: Added LNI 192.168.203.12@tcp231 [8/256/0/180] [ 694.487943] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:40:33 (1771440033) [ 694.532505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.546609] LNet: Removed LNI 192.168.203.12@tcp231 [ 694.554557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.576526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 694.585946] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 694.826595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 694.842500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 694.864545] LNet: Removed LNI 192.168.203.12@tcp232 [ 694.870807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 694.883200] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 695.124520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 695.139981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.158285] LNet: Removed LNI 192.168.203.12@tcp232 [ 695.165371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 695.175758] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 695.414821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 695.429493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.449296] LNet: Removed LNI 192.168.203.12@tcp232 [ 695.455380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 695.464780] LNet: Added LNI 192.168.203.12@tcp232 [8/256/0/180] [ 695.717401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 695.730926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.748450] LNet: Removed LNI 192.168.203.12@tcp232 [ 695.755111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 697.858132] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:40:36 (1771440036) [ 698.224046] Key type .llcrypt unregistered [ 698.225496] Key type ._llcrypt unregistered [ 700.189379] Key type ._llcrypt registered [ 700.190465] Key type .llcrypt registered [ 700.255676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 700.261723] alg: No test for adler32 (adler32-zlib) [ 701.063647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 701.075150] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 703.960391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 706.922650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.938893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 706.948615] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 707.517877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_interrupt -r 2 [ 707.533431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 2 [ 707.550651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_interrupt -r 2 [ 707.566332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 2 [ 707.583531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_interrupt -r 2 [ 707.602412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_interrupt -r 2 [ 707.618412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_interrupt -r 2 [ 707.633478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_interrupt -r 2 [ 707.645607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 707.655527] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 707.658624] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 707.744672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 707.759977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 707.774981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 707.792901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 707.809790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 707.823476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 707.836858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 707.850745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 707.859836] LNet: 87053:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 707.869821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 707.885045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 707.898719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 707.912725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 707.926823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 707.942442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 708.146927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_dropped -r 2 [ 708.163533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 2 [ 708.182412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_dropped -r 2 [ 708.197509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 2 [ 708.218476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_dropped -r 2 [ 708.236471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_dropped -r 2 [ 708.252984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_dropped -r 2 [ 708.272281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_dropped -r 2 [ 708.288915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 708.299433] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 708.383723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 708.396531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 708.408833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 708.421671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 708.434898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 708.449420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 708.466971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 708.481967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 708.491120] LNet: 87174:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 708.502657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 708.518773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 708.532523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 708.547098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 708.560662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 708.575598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 708.768690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_aborted -r 2 [ 708.784277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 2 [ 708.801191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_aborted -r 2 [ 708.818185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 2 [ 708.836674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_aborted -r 2 [ 708.853466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_aborted -r 2 [ 708.869915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_aborted -r 2 [ 708.884944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_aborted -r 2 [ 708.898438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 708.992298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 709.006185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 709.021280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 709.034965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 709.049269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 709.065296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 709.078593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 709.092826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 709.105990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 709.120505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 709.134422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 709.148263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 709.162073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 709.174533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 709.375985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_no_route -r 2 [ 709.392866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 2 [ 709.410339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_no_route -r 2 [ 709.426943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 2 [ 709.445498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_no_route -r 2 [ 709.466773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_no_route -r 2 [ 709.487913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_no_route -r 2 [ 709.505019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_no_route -r 2 [ 709.522242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 709.532945] LNet: There was an unexpected network error while writing to 192.168.203.112: rc = -22 [ 709.535974] LNet: Skipped 2 previous similar messages [ 709.621373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 709.633591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 709.653315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 709.667763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 709.686141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 709.702182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 709.718756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 709.734548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 709.743732] LNet: 87414:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 709.746710] LNet: 87414:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 709.757550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 709.773729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 709.788878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 709.804658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 709.822485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 709.837599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 710.052941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e local_timeout -r 2 [ 710.073963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 2 [ 710.090968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e local_timeout -r 2 [ 710.109782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 2 [ 710.132455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e local_timeout -r 2 [ 710.153156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e local_timeout -r 2 [ 710.171661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e local_timeout -r 2 [ 710.188419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e local_timeout -r 2 [ 710.205497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 710.307680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 710.322646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 710.337973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 710.353918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 710.371250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 710.384897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 710.402770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 710.421601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 710.437353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 710.453603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 710.473372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 710.491706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 710.508226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 710.524463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 710.727771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp -m GET -e remote_dropped -r 2 [ 710.747368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 2 [ 710.767626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp -m GET -e remote_dropped -r 2 [ 710.784214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 2 [ 710.802636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp -m GET -e remote_dropped -r 2 [ 710.821962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.112@tcp1 -m GET -e remote_dropped -r 2 [ 710.842701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp -m GET -e remote_dropped -r 2 [ 710.866976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.12@tcp1 -d 192.168.203.12@tcp1 -m GET -e remote_dropped -r 2 [ 710.885556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.12@tcp 192.168.203.112@tcp [ 710.899722] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 710.995550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 711.013883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 711.027059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 711.042924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 711.058350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 711.073663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 711.088727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 711.102183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 711.114225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 711.127447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 711.141339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 711.156589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 711.170410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 711.183608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 713.464938] LNet: 87727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.468456] LNetError: 87727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.471947] LNet: Removed LNI 192.168.203.12@tcp [ 713.481481] LNet: Removed LNI 192.168.203.12@tcp1 [ 713.641427] Key type .llcrypt unregistered [ 713.643215] Key type ._llcrypt unregistered [ 718.706453] Key type ._llcrypt registered [ 718.707753] Key type .llcrypt registered [ 719.164585] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:40:57 (1771440057) [ 719.293982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 719.301159] alg: No test for adler32 (adler32-zlib) [ 720.114538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.138434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.148096] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 720.442000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 720.454067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.12@tcp [ 720.461854] LNetError: 88703:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 720.492929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.12@tcp [ 720.532421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 722.729659] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:41:01 (1771440061) [ 722.764155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.773535] LNet: 89289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.778110] LNetError: 89289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.789616] LNet: Removed LNI 192.168.203.12@tcp [ 722.799106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.818904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.828549] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 723.094832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 723.104378] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 723.460681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.12@tcp [ 723.469562] LNetError: 89558:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.203.12@tcp to recovery [ 723.808309] LNet: 1 local NIs in recovery (showing 1): 192.168.203.12@tcp [ 723.808921] LNet: There was an unexpected network error while writing to 192.168.203.12: rc = -22 [ 724.548100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.12@tcp [ 728.752199] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:41:07 (1771440067) [ 729.351345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg312-server [ 729.893487] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:41:08 (1771440068) [ 730.509075] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg312-server [ 731.040783] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:41:09 (1771440069) [ 731.628144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg312-server [ 732.147462] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:41:10 (1771440070) [ 732.184769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.193696] LNet: 91183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.195987] LNetError: 91183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.197937] LNet: Removed LNI 192.168.203.12@tcp [ 732.206908] LNet: Removed LNI 192.168.203.12@tcp1 [ 732.214980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.232523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.239458] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 732.463577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 732.484876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 732.501350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 732.527756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 732.544407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 732.561345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 734.584853] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:41:13 (1771440073) [ 734.637423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.649269] LNet: 91910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.652255] LNetError: 91910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.655432] LNetError: 91910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 734.664406] LNet: Removed LNI 192.168.203.12@tcp [ 734.672481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.693184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.702123] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 736.947141] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:41:15 (1771440075) [ 736.990786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.999659] LNet: 92610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.003115] LNetError: 92610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.018755] LNet: Removed LNI 192.168.203.12@tcp [ 737.027432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.049285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.057782] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 739.333291] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:41:17 (1771440077) [ 739.874598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 740.471751] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:41:18 (1771440078) [ 740.624412] LNet: Removed LNI 192.168.203.12@tcp [ 740.833608] Key type .llcrypt unregistered [ 740.834765] Key type ._llcrypt unregistered [ 742.783754] Key type ._llcrypt registered [ 742.784767] Key type .llcrypt registered [ 742.847504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.854694] alg: No test for adler32 (adler32-zlib) [ 743.655736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 743.667462] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 746.534703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 748.154541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.112@tcp [ 748.260650] LNet: 94062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.263167] LNetError: 94062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.278308] LNet: Removed LNI 192.168.203.12@tcp [ 748.421394] Key type .llcrypt unregistered [ 748.422749] Key type ._llcrypt unregistered [ 752.339333] Key type ._llcrypt registered [ 752.340286] Key type .llcrypt registered [ 752.844221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 753.346978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 753.898627] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:41:32 (1771440092) [ 754.023785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.028957] alg: No test for adler32 (adler32-zlib) [ 754.834464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.863331] LNet: 95147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.077265] Key type .llcrypt unregistered [ 755.078849] Key type ._llcrypt unregistered [ 757.098218] Key type ._llcrypt registered [ 757.099253] Key type .llcrypt registered [ 757.156868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.162916] alg: No test for adler32 (adler32-zlib) [ 757.970777] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 757.972041] LNet: 95402:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 757.974569] LNet: 95402:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 758.137971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.060804] LNetError: 95409:0:(lib-move.c:4960:lnet_parse()) 192.168.203.17@tcp, src 192.168.203.17@tcp: Dropping REPLY (error -108 looking up sender) [ 761.248114] LNetError: 95414:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.14@tcp has gone from up to down [ 761.253320] LNet: 95460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.256253] LNetError: 95460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.269356] LNet: Removed LNI 192.168.203.12@tcp [ 761.277140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.296290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 761.304259] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 761.306317] LNet: 95468:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 761.308843] LNet: 95468:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 761.311916] LNetError: 95468:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.13@tcp is a gateway. Can not delete it [ 761.314446] LNetError: 95468:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.203.13@tcp is a router. Can not be deleted [ 761.317310] LNet: 95468:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 763.619897] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:41:42 (1771440102) [ 764.128534] LNetError: 95470:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.14@tcp has gone from up to down [ 764.133311] LNetError: 95470:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 6 previous similar messages [ 764.138900] LNetError: 95465:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 764.210359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg312-server [ 764.792454] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:41:43 (1771440103) [ 765.427841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg312-server [ 766.001700] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:41:44 (1771440104) [ 766.589680] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 768.256148] Key type lgssc registered [ 768.620058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 776.499457] Key type lgssc unregistered [ 776.663654] LNet: 97884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.667292] LNetError: 97884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.681519] LNet: Removed LNI 192.168.203.12@tcp [ 776.895594] Key type .llcrypt unregistered [ 776.896664] Key type ._llcrypt unregistered [ 781.864205] Key type ._llcrypt registered [ 781.865226] Key type .llcrypt registered [ 782.322159] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:42:00 (1771440120) [ 782.454094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.460038] alg: No test for adler32 (adler32-zlib) [ 783.265288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.291916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 783.311839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 783.330931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 783.349575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 785.425732] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:42:03 (1771440123) [ 785.465687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.474139] LNet: 99286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.485972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.508914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.519192] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 785.834403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.843312] LNetError: 99432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.858300] LNet: Removed LNI 192.168.203.12@tcp [ 785.865647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.883262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 785.892110] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 786.061478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.068142] LNet: 99460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.070232] LNet: 99460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 786.079360] LNet: Removed LNI 192.168.203.12@tcp [ 786.086209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.107694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 786.117762] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 788.335121] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:42:06 (1771440126) [ 788.381767] LNet: 100045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.384730] LNetError: 100045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.387404] LNetError: 100045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 788.397579] LNet: Removed LNI 192.168.203.12@tcp [ 788.632637] Key type .llcrypt unregistered [ 788.637315] Key type ._llcrypt unregistered [ 790.658853] Key type ._llcrypt registered [ 790.660419] Key type .llcrypt registered [ 790.728511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.733528] alg: No test for adler32 (adler32-zlib) [ 791.536547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.564874] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 791.567139] LNet: *** cfs_fail_loc=e003, val=100*** [ 791.831389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 791.841135] Ping reply received with mapping: [ 791.842708] found metadata reply NI[192.168.203.12@tcp] [ 791.845270] 0c cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 791.848191] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 791.849926] 5a 5a 5a 5a 5a 5a 5a 5a [ 791.899501] LNet: 100447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.901587] LNetError: 100447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.913279] LNet: Removed LNI 192.168.203.12@tcp [ 792.129327] Key type .llcrypt unregistered [ 792.130680] Key type ._llcrypt unregistered [ 795.711351] Key type ._llcrypt registered [ 795.712364] Key type .llcrypt registered [ 796.144931] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:42:14 (1771440134) [ 796.302532] Key type .llcrypt unregistered [ 796.303937] Key type ._llcrypt unregistered [ 798.279247] Key type ._llcrypt registered [ 798.280554] Key type .llcrypt registered [ 798.346811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.353293] alg: No test for adler32 (adler32-zlib) [ 799.214376] LNetError: 101490:0:(api-ni.c:2734:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 799.345338] Key type .llcrypt unregistered [ 799.346379] Key type ._llcrypt unregistered [ 800.930883] Key type ._llcrypt registered [ 800.931989] Key type .llcrypt registered [ 801.370390] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:42:19 (1771440139) [ 801.549230] Key type .llcrypt unregistered [ 801.550349] Key type ._llcrypt unregistered [ 803.371196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.373188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.527818] Key type ._llcrypt registered [ 805.528806] Key type .llcrypt registered [ 805.588274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.593533] alg: No test for adler32 (adler32-zlib) [ 806.390357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.410634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.048421] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:42:28 (1771440148) [ 810.079949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.087270] LNet: 103263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.089865] LNetError: 103263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.095805] LNet: Removed LNI 10.1.2.3@tcp [ 810.106260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.337421] Key type .llcrypt unregistered [ 810.338339] Key type ._llcrypt unregistered [ 812.231306] Key type ._llcrypt registered [ 812.232580] Key type .llcrypt registered [ 812.288386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.293614] alg: No test for adler32 (adler32-zlib) [ 813.098669] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 813.122482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 813.174077] LNet: 103555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.176401] LNetError: 103555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.187279] LNet: Removed LNI 192.168.203.12@tcp [ 813.405397] Key type .llcrypt unregistered [ 813.406539] Key type ._llcrypt unregistered [ 816.989516] Key type ._llcrypt registered [ 816.990510] Key type .llcrypt registered [ 817.433114] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:42:35 (1771440155) [ 817.643393] Key type .llcrypt unregistered [ 817.644517] Key type ._llcrypt unregistered [ 819.596031] Key type ._llcrypt registered [ 819.597765] Key type .llcrypt registered [ 819.663649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.668813] alg: No test for adler32 (adler32-zlib) [ 822.502454] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:42:41 (1771440161) [ 822.548892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.564467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.592604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.604051] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 822.856274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.868664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 822.884581] LNet: Removed LNI 192.168.203.12@tcp [ 822.957742] LNet: 105309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.093280] Key type .llcrypt unregistered [ 823.094622] Key type ._llcrypt unregistered [ 826.610068] Key type ._llcrypt registered [ 826.611550] Key type .llcrypt registered [ 827.051990] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:42:45 (1771440165) [ 827.178521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.183936] alg: No test for adler32 (adler32-zlib) [ 827.987682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.012162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.021972] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 828.278411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 828.292427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.12@tcp [ 828.299471] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 828.301358] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 828.303125] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 828, 0, 0 [ 828.305312] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 828.306460] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 828.307981] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 828.309657] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 828, 0, 0 [ 828.311508] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 828.312701] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 828.314425] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 828.315974] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 828, 0, 878 [ 828.317766] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 828.319181] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 828.320560] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 828.321716] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 828, 0, 878 [ 828.323482] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 830.417373] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:42:48 (1771440168) [ 830.567572] LNet: 106846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.569567] LNetError: 106846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.578463] LNet: Removed LNI 192.168.203.12@tcp [ 830.803493] Key type .llcrypt unregistered [ 830.804792] Key type ._llcrypt unregistered [ 832.834569] Key type ._llcrypt registered [ 832.835926] Key type .llcrypt registered [ 832.906766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.913325] alg: No test for adler32 (adler32-zlib) [ 833.715827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 833.726910] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 836.557850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 839.141599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.153109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 839.161508] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 839.460752] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.490729] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.493085] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.495662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 841.627950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 841.636912] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 844.910085] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:43:03 (1771440183) [ 844.943466] LNet: 108343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.946251] LNetError: 108343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.949279] LNet: Removed LNI 192.168.203.12@tcp [ 845.984146] LNet: Removed LNI 192.168.203.12@tcp1 [ 845.985450] LNetError: 108343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 845.988014] LNetError: 108343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.996472] LNet: Removed LNI 10.1.2.3@tcp99 [ 846.210512] Key type .llcrypt unregistered [ 846.211625] Key type ._llcrypt unregistered [ 848.088566] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.090240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.248676] Key type ._llcrypt registered [ 850.249578] Key type .llcrypt registered [ 850.308502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.313584] alg: No test for adler32 (adler32-zlib) [ 851.112778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.133602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.143364] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 851.396878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 851.404270] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.116361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 852.134324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 852.148866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 852.156621] LNetError: 109092:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 852.169584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 852.186274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 852.326097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 852.352162] LNet: 109112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.354191] LNetError: 109112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.356865] LNet: Removed LNI 192.168.203.12@tcp [ 852.368320] LNet: Removed LNI 10.1.2.3@tcp [ 852.590403] Key type .llcrypt unregistered [ 852.592035] Key type ._llcrypt unregistered [ 856.012412] Key type ._llcrypt registered [ 856.013339] Key type .llcrypt registered [ 856.440623] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:43:14 (1771440194) [ 856.560146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.564919] alg: No test for adler32 (adler32-zlib) [ 857.369767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.389953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.398967] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 857.642338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.12@tcp [ 857.669743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg312-client@tcp [ 859.619228] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:43:18 (1771440198) [ 859.651058] LNet: 110637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.654361] LNetError: 110637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.660473] LNet: Removed LNI 192.168.203.12@tcp [ 859.870972] Key type .llcrypt unregistered [ 859.871880] Key type ._llcrypt unregistered [ 861.848171] Key type ._llcrypt registered [ 861.849430] Key type .llcrypt registered [ 861.904753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.910303] alg: No test for adler32 (adler32-zlib) [ 862.704225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 862.714216] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 862.728907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 864.780914] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:43:23 (1771440203) [ 865.262357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 865.802491] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:43:24 (1771440204) [ 866.306523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 866.846323] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:43:25 (1771440205) [ 866.887942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.899191] LNet: 112057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.902635] LNetError: 112057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.910696] LNet: Removed LNI 192.168.203.12@tcp [ 866.918600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.940240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.949811] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 867.205208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 867.225089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 869.430052] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:43:27 (1771440207) [ 869.463976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.474748] LNet: 112762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.477872] LNetError: 112762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.488634] LNet: Removed LNI 192.168.203.12@tcp [ 869.500760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.525406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 869.538884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 869.786538] Key type .llcrypt unregistered [ 869.788044] Key type ._llcrypt unregistered [ 873.563754] Key type ._llcrypt registered [ 873.565044] Key type .llcrypt registered [ 874.094643] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:43:32 (1771440212) [ 874.234789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.242022] alg: No test for adler32 (adler32-zlib) [ 875.057082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.078775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.095795] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 875.391548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 875.418456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 877.619247] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:43:36 (1771440216) [ 877.654190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.664358] LNet: 114316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.667277] LNetError: 114316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.676539] LNet: Removed LNI 192.168.203.12@tcp [ 877.687724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.712215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 877.727322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 879.717875] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:43:38 (1771440218) [ 879.756648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.765458] LNet: 114898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.774945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.798884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 879.930400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 879.947310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 879.972379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 880.099959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 880.114434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 880.141158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.154910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.179152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 880.311207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 880.321806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 880.341467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 880.471700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 880.483539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 882.590570] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:43:41 (1771440221) [ 882.636718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.648966] LNet: 115532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.651529] LNet: 115532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 882.661715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.684630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.692554] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 882.948458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 882.956277] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 883.260558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 883.283636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 883.364467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.373421] LNetError: 115812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.376116] LNet: Removed LNI 192.168.203.12@tcp [ 883.381577] LNet: Removed LNI 192.168.203.12@tcp2 [ 883.390376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.416872] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 883.419953] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 883.423166] LNet: 115818:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 883.615353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 883.680925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.704122] LNet: Removed LNI 192.168.203.12@tcp [ 884.705381] LNetError: 115855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.707301] LNetError: 115855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 884.715303] LNet: Removed LNI 192.168.203.12@tcp2 [ 884.723454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.743368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.752824] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 885.017562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 885.027060] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 885.341120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 885.357236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 885.376765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 885.447584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.455798] LNet: 116139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.458078] LNet: 116139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 886.496119] LNet: Removed LNI 192.168.203.12@tcp [ 887.526652] LNet: Removed LNI 192.168.203.12@tcp2 [ 887.536124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.560185] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 887.562510] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 887.564601] LNet: 116146:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 887.730660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 887.796578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.806354] LNetError: 116183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.809296] LNetError: 116183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 887.812474] LNet: Removed LNI 192.168.203.12@tcp [ 888.870238] LNet: Removed LNI 192.168.203.12@tcp2 [ 888.877118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.894988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.902490] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 889.132712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 889.139082] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 889.410476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 889.427987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 889.495861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.503972] LNet: 116462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.506898] LNet: 116462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 890.528131] LNet: Removed LNI 192.168.203.12@tcp [ 890.535328] LNet: Removed LNI 192.168.203.12@tcp2 [ 890.542859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.566973] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 890.569577] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 890.571824] LNet: 116468:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 890.736588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 890.796757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.808138] LNet: Removed LNI 192.168.203.12@tcp [ 891.809710] LNetError: 116505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.812157] LNetError: 116505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 891.821249] LNet: Removed LNI 192.168.203.12@tcp2 [ 891.827964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.847033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.855531] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 892.106314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 892.115407] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 892.411383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 892.425095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 892.443743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 892.512865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.536130] LNet: Removed LNI 192.168.203.12@tcp [ 894.566283] LNet: Removed LNI 192.168.203.12@tcp2 [ 894.573179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.593625] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 894.595842] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 894.597850] LNet: 116796:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 894.749661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 894.813451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.823778] LNet: Removed LNI 192.168.203.12@tcp [ 895.846418] LNet: Removed LNI 192.168.203.12@tcp2 [ 895.855452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.879310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.888907] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 896.141414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 896.149771] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 896.468121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 896.479791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 896.501655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 896.630266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.696103] LNet: Removed LNI 192.168.203.12@tcp [ 897.703299] LNet: Removed LNI 192.168.203.12@tcp2 [ 897.711693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.735086] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 897.737446] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 897.900884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 898.009869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.017606] LNet: 117187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.019595] LNet: 117187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 899.040126] LNet: Removed LNI 192.168.203.12@tcp [ 899.047616] LNet: Removed LNI 192.168.203.12@tcp2 [ 899.055513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.077518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.087312] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 899.367574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 899.375981] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 899.714913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 899.732992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 899.748678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 899.775177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 899.900681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.909162] LNetError: 117488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.912130] LNetError: 117488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 899.914576] LNet: Removed LNI 192.168.203.12@tcp [ 900.966367] LNet: Removed LNI 192.168.203.12@tcp2 [ 900.975779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.000842] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 901.003600] LNet: Added LNI 192.168.203.12@tcp2 [8/256/0/180] [ 901.006116] LNet: 117495:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.008739] LNet: 117495:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 901.178025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 903.556671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 904.101777] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:44:02 (1771440242) [ 904.137326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.149530] LNet: Removed LNI 192.168.203.12@tcp [ 905.190429] LNet: Removed LNI 192.168.203.12@tcp2 [ 905.198373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.295295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.303736] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 905.542431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 905.551890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 905.855185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.864149] LNet: Removed LNI 192.168.203.12@tcp [ 905.874264] LNet: Removed LNI 10.1.2.3@tcp [ 905.885550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.909971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 905.920350] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 905.922571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.069189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 906.079596] LNet: Removed LNI 192.168.203.12@tcp [ 906.095270] LNet: Removed LNI 10.1.2.3@tcp [ 908.188863] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:44:06 (1771440246) [ 908.247948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.265623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.305869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 908.380611] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 908.382915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 908.385098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 910.497282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.505501] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 910.781327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 910.792977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.488754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.12@tcp [ 911.505433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 911.528216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.12@tcp -m PUT -e local_timeout -r 1 [ 911.550994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 911.572333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.575045] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.654783] LNetError: 119192:0:(lib-move.c:4960:lnet_parse()) 192.168.203.12@tcp, src 192.168.203.12@tcp: Dropping PUT (error -108 looking up sender) [ 913.696158] LNet: Removed LNI 192.168.203.12@tcp [ 913.704481] LNet: Removed LNI 10.1.2.3@tcp [ 913.842884] Key type .llcrypt unregistered [ 913.844054] Key type ._llcrypt unregistered [ 917.659065] Key type ._llcrypt registered [ 917.660421] Key type .llcrypt registered [ 918.069818] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:44:16 (1771440256) [ 918.192255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.197640] alg: No test for adler32 (adler32-zlib) [ 918.996791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.115081] LNet: 120511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.324213] Key type .llcrypt unregistered [ 919.325112] Key type ._llcrypt unregistered [ 921.247906] Key type ._llcrypt registered [ 921.249292] Key type .llcrypt registered [ 921.315690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.321880] alg: No test for adler32 (adler32-zlib) [ 922.127418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 922.138238] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 924.964674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 925.014608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 925.208567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.112@tcp [ 925.286693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 925.300632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 925.314634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 925.327512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 925.339777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 925.351921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 925.364046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 925.375853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 925.383319] LNet: 120959:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 925.390090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 925.402236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 925.413827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 925.426742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 925.438947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 925.450366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 925.659965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 925.749742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 925.761678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 925.774051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 925.785580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 925.799986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 925.813373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 925.830829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 925.845330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 925.860056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 925.874559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 925.890950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 925.908386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 925.921966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 925.936168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.173719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.112@tcp [ 926.241201] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 926.261109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.273343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 926.285910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.299391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.311467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.332015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.346972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.362288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.370426] LNet: 121193:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.373221] LNet: 121193:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 926.380573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.394313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.411297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.423692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.437767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.453112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.694966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 926.792152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.806430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 926.823792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.838523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.853655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.867984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.883826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.899515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.914828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.931757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.949779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.966342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.982515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.998844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.137126] LNet: 121357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.140737] LNetError: 121357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.151423] LNet: Removed LNI 192.168.203.12@tcp [ 927.328320] Key type .llcrypt unregistered [ 927.329765] Key type ._llcrypt unregistered [ 933.170566] Key type ._llcrypt registered [ 933.171976] Key type .llcrypt registered [ 933.671073] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:44:32 (1771440272) [ 933.789051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.793930] alg: No test for adler32 (adler32-zlib) [ 934.599338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.724055] LNet: 122242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.939480] Key type .llcrypt unregistered [ 934.940823] Key type ._llcrypt unregistered [ 936.974783] Key type ._llcrypt registered [ 936.976330] Key type .llcrypt registered [ 937.058055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.065230] alg: No test for adler32 (adler32-zlib) [ 937.868401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 937.879291] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 940.788978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.112@tcp [ 943.537246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.550285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 943.559978] LNet: Added LNI 192.168.203.12@tcp1 [8/256/0/180] [ 943.879284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 944.068850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.112@tcp [ 944.145215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.155365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 944.165573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.177251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.188234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.199571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.211375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.223908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.231043] LNet: 122891:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 944.237668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.251071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.262956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.275393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.288166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.302627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.501159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 944.578931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.591888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 944.605865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.618928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.631662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.644273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.656523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.668972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.681154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.692453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.703156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.714333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.725779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.738594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.940172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.112@tcp [ 945.023202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.036390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 945.047663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.056184] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.112@tcp [ 945.059650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.072187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.084064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.096084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.107856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.114751] LNet: 123125:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 945.117316] LNet: 123125:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 945.125182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.138166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.150084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.161699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.173230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.183613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.417235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 945.505303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.518729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 945.532803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.547172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.561724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.576710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.592363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.607727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.622172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.636872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.650215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.664558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.677976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.691470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 947.884648] LNet: 123323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.888259] LNetError: 123323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.893044] LNet: Removed LNI 192.168.203.12@tcp [ 947.900330] LNet: Removed LNI 192.168.203.12@tcp1 [ 948.046616] Key type .llcrypt unregistered [ 948.047661] Key type ._llcrypt unregistered [ 953.554115] Key type ._llcrypt registered [ 953.555194] Key type .llcrypt registered [ 954.003504] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:44:52 (1771440292) [ 956.421171] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:44:54 (1771440294) [ 958.716883] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:44:57 (1771440297) [ 961.146377] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:44:59 (1771440299) [ 963.529099] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:45:02 (1771440302) [ 965.936573] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:45:04 (1771440304) [ 968.376565] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:45:06 (1771440306) [ 970.660973] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:45:09 (1771440309) [ 972.962424] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:45:11 (1771440311) [ 975.269896] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:45:13 (1771440313) [ 977.577838] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:45:16 (1771440316) [ 979.853819] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:45:18 (1771440318) [ 982.169792] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:45:20 (1771440320) [ 984.253648] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:45:22 (1771440322) [ 986.470379] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:45:25 (1771440325) [ 988.803479] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:45:27 (1771440327) [ 991.056922] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:45:29 (1771440329) [ 993.357181] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:45:31 (1771440331) [ 993.476891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.482897] alg: No test for adler32 (adler32-zlib) [ 994.286889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.330207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.339694] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 994.713835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.723936] LNet: 134020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.726532] LNetError: 134020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.736292] LNet: Removed LNI 192.168.203.12@tcp [ 994.743830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.765052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 994.774437] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 997.047219] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:45:35 (1771440335) [ 999.416818] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:45:37 (1771440337) [ 1001.591683] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:45:40 (1771440340) [ 1003.876714] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:45:42 (1771440342) [ 1006.092344] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:45:44 (1771440344) [ 1008.257464] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:45:46 (1771440346) [ 1010.440233] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:45:48 (1771440348) [ 1012.732489] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:45:51 (1771440351) [ 1014.798792] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:45:53 (1771440353) [ 1016.871652] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:45:55 (1771440355) [ 1018.923557] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:45:57 (1771440357) [ 1021.014233] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:45:59 (1771440359) [ 1023.121677] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 899 sec =========== 13:46:01 (1771440361) [ 1023.211484] LNet: 141474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.213470] LNetError: 141474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.221809] LNet: Removed LNI 192.168.203.12@tcp [ 1023.423498] Key type .llcrypt unregistered [ 1023.424886] Key type ._llcrypt unregistered [ 1027.043545] Key type ._llcrypt registered [ 1027.044357] Key type .llcrypt registered [ 1027.204829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.209923] alg: No test for adler32 (adler32-zlib) [ 1028.058244] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1028.143479] LNet: Added LNI 192.168.203.12@tcp [8/256/0/180] [ 1029.728118] Key type lgssc registered [ 1030.100742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.364175] Lustre: Mounted lustre-client [ 1053.699293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1060.865301] LustreError: 143919:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d3bc868d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1060.881784] Lustre: Unmounted lustre-client [ 1072.427638] Key type lgssc unregistered [ 1072.544531] LNet: 144401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.547264] LNetError: 144401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.555651] LNet: Removed LNI 192.168.203.12@tcp [ 1072.786143] Key type .llcrypt unregistered [ 1072.787304] Key type ._llcrypt unregistered