[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 435675125 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003175] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008413] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.021714] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025737] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030051] *** VALIDATE tmpfs *** [ 0.031361] *** VALIDATE proc *** [ 0.032144] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042184] debug: unmapping init [mem 0xffffffff86659000-0xffffffff86660fff] [ 0.044205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045565] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053277] rcu: Hierarchical SRCU implementation. [ 0.055290] smp: Bringing up secondary CPUs ... [ 0.056480] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.064139] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143019] node 0 deferred pages initialised in 74ms [ 0.146105] devtmpfs: initialized [ 0.147290] x86/mm: Memory block size: 128MB [ 0.150982] gcov: version magic: 0x41383552 [ 0.154278] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159280] pinctrl core: initialized pinctrl subsystem [ 0.161169] [ 0.161836] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166009] ** ** [ 0.168013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170010] ** ** [ 0.172010] ** This means that this kernel is built to expose internal ** [ 0.175013] ** IOMMU data structures, which may compromise security on ** [ 0.177010] ** your system. ** [ 0.179012] ** ** [ 0.181014] ** If you see this message and you are not debugging the ** [ 0.184012] ** kernel, report this immediately to your vendor! ** [ 0.186011] ** ** [ 0.189013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192013] ************************************************************* [ 0.194553] NET: Registered protocol family 16 [ 0.196398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206028] cpuidle: using governor menu [ 0.207944] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210617] PCI: Using configuration type 1 for base access [ 0.214154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230204] cryptd: max_cpu_qlen set to 1000 [ 0.234013] ACPI: Added _OSI(Module Device) [ 0.235025] ACPI: Added _OSI(Processor Device) [ 0.237019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244295] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249466] ACPI: Interpreter enabled [ 0.250055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251013] ACPI: Using IOAPIC for interrupt routing [ 0.253116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267700] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276147] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281325] acpiphp: Slot [2] registered [ 0.283108] acpiphp: Slot [5] registered [ 0.285337] acpiphp: Slot [6] registered [ 0.286102] acpiphp: Slot [3] registered [ 0.288061] acpiphp: Slot [4] registered [ 0.289135] acpiphp: Slot [7] registered [ 0.290126] acpiphp: Slot [8] registered [ 0.292113] acpiphp: Slot [9] registered [ 0.293116] acpiphp: Slot [10] registered [ 0.295105] acpiphp: Slot [11] registered [ 0.296106] acpiphp: Slot [12] registered [ 0.298130] acpiphp: Slot [13] registered [ 0.299067] acpiphp: Slot [14] registered [ 0.301103] acpiphp: Slot [15] registered [ 0.302110] acpiphp: Slot [16] registered [ 0.304118] acpiphp: Slot [17] registered [ 0.306105] acpiphp: Slot [18] registered [ 0.307133] acpiphp: Slot [19] registered [ 0.309107] acpiphp: Slot [20] registered [ 0.310098] acpiphp: Slot [21] registered [ 0.312074] acpiphp: Slot [22] registered [ 0.313084] acpiphp: Slot [23] registered [ 0.314093] acpiphp: Slot [24] registered [ 0.316119] acpiphp: Slot [25] registered [ 0.317105] acpiphp: Slot [26] registered [ 0.318083] acpiphp: Slot [27] registered [ 0.319067] acpiphp: Slot [28] registered [ 0.320082] acpiphp: Slot [29] registered [ 0.322111] acpiphp: Slot [30] registered [ 0.323146] acpiphp: Slot [31] registered [ 0.325060] PCI host bridge to bus 0000:00 [ 0.326018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341094] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345393] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352007] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.356466] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.359017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366056] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368938] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.390015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.394014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400397] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444540] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489440] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.493388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.496369] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.498418] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.500238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.506046] iommu: Default domain type: Passthrough [ 0.508581] SCSI subsystem initialized [ 0.510180] ACPI: bus type USB registered [ 0.512145] usbcore: registered new interface driver usbfs [ 0.514161] usbcore: registered new interface driver hub [ 0.516148] usbcore: registered new device driver usb [ 0.518242] pps_core: LinuxPPS API ver. 1 registered [ 0.520017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.523088] PTP clock support registered [ 0.526073] EDAC MC: Ver: 3.0.0 [ 0.528446] PCI: Using ACPI for IRQ routing [ 0.531014] NetLabel: Initializing [ 0.532014] NetLabel: domain hash size = 128 [ 0.534013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.537117] NetLabel: unlabeled traffic allowed by default [ 0.539205] vgaarb: loaded [ 0.541316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.543025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.550222] clocksource: Switched to clocksource kvm-clock [ 0.657410] VFS: Disk quotas dquot_6.6.0 [ 0.659308] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.662164] *** VALIDATE ramfs *** [ 0.663574] *** VALIDATE hugetlbfs *** [ 0.665449] pnp: PnP ACPI init [ 0.668110] pnp: PnP ACPI: found 6 devices [ 0.693268] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.698546] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700824] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.703412] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.706078] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.708144] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.711083] NET: Registered protocol family 2 [ 0.713462] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.717489] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.721032] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726847] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730528] TCP: Hash tables configured (established 65536 bind 65536) [ 0.733556] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.736655] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.739318] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.741862] NET: Registered protocol family 1 [ 0.744860] RPC: Registered named UNIX socket transport module. [ 0.747158] RPC: Registered udp transport module. [ 0.748855] RPC: Registered tcp transport module. [ 0.751813] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.755116] NET: Registered protocol family 44 [ 0.756830] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.759479] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.761747] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.764395] PCI: CLS 0 bytes, default 64 [ 0.766088] Unpacking initramfs... [ 2.220571] debug: unmapping init [mem 0xffff9e98bcc64000-0xffff9e98bffcffff] [ 2.226962] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.229336] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.234051] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.750202] Initialise system trusted keyrings [ 2.752268] Key type blacklist registered [ 2.757633] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.770072] zbud: loaded [ 2.774562] *** VALIDATE nfs *** [ 2.775948] *** VALIDATE nfs4 *** [ 2.777642] pstore: using deflate compression [ 2.782703] Platform Keyring initialized [ 2.892537] NET: Registered protocol family 38 [ 2.894345] Key type asymmetric registered [ 2.895450] Asymmetric key parser 'x509' registered [ 2.896963] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.899325] io scheduler mq-deadline registered [ 2.900704] io scheduler kyber registered [ 2.901917] io scheduler bfq registered [ 2.903251] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.905523] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.907820] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.910023] ACPI: Power Button [PWRF] [ 2.916107] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.921780] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.932595] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.959573] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.988829] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.995784] Non-volatile memory driver v1.3 [ 2.997755] Linux agpgart interface v0.103 [ 3.035740] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.040563] vda: detected capacity change from 0 to 68575232 [ 3.055344] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.058405] vdb: detected capacity change from 0 to 1073741824 [ 3.065928] libphy: Fixed MDIO Bus: probed [ 3.071823] usbcore: registered new interface driver usbserial_generic [ 3.074497] usbserial: USB Serial support registered for generic [ 3.076753] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.080273] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081650] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.083664] mousedev: PS/2 mouse device common for all mice [ 3.086739] rtc_cmos 00:05: RTC can wake from S4 [ 3.088866] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.089207] rtc_cmos 00:05: registered as rtc0 [ 3.092711] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.095288] intel_pstate: CPU model not supported [ 3.097631] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.099092] hid: raw HID events driver (C) Jiri Kosina [ 3.103151] usbcore: registered new interface driver usbhid [ 3.103257] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.104709] usbhid: USB HID core driver [ 3.104843] drop_monitor: Initializing network drop monitor service [ 3.111147] Initializing XFRM netlink socket [ 3.112649] NET: Registered protocol family 10 [ 3.114910] Segment Routing with IPv6 [ 3.116472] NET: Registered protocol family 17 [ 3.118628] mpls_gso: MPLS GSO support [ 3.124438] RAS: Correctable Errors collector initialized. [ 3.126825] AVX version of gcm_enc/dec engaged. [ 3.128804] AES CTR mode by8 optimization enabled [ 3.207573] sched_clock: Marking stable (3207555063, 0)->(4063757435, -856202372) [ 3.210427] registered taskstats version 1 [ 3.211988] Loading compiled-in X.509 certificates [ 3.213542] zswap: loaded using pool lzo/zbud [ 3.236256] Key type big_key registered [ 3.247404] Key type encrypted registered [ 3.250532] ima: No TPM chip found, activating TPM-bypass! [ 3.252499] ima: Allocated hash algorithm: sha1 [ 3.254055] ima: No architecture policies found [ 3.255416] evm: Initialising EVM extended attributes: [ 3.257501] evm: security.selinux [ 3.258676] evm: security.ima [ 3.259765] evm: security.capability [ 3.261099] evm: HMAC attrs: 0x1 [ 3.263348] rtc_cmos 00:05: setting system clock to 2026-04-03 11:42:36 UTC (1775216556) [ 3.269411] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.272726] debug: unmapping init [mem 0xffffffff86382000-0xffffffff86658fff] [ 3.280154] Write protecting the kernel read-only data: 28672k [ 3.282695] debug: unmapping init [mem 0xffffffff84a03000-0xffffffff84bfffff] [ 3.285041] debug: unmapping init [mem 0xffffffff85314000-0xffffffff853fffff] [ 3.316753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.325994] systemd[1]: Detected virtualization kvm. [ 3.328238] systemd[1]: Detected architecture x86-64. [ 3.330283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.358695] systemd[1]: No hostname configured. [ 3.360613] systemd[1]: Set hostname to . [ 3.362763] random: systemd: uninitialized urandom read (16 bytes read) [ 3.365431] systemd[1]: Initializing machine ID from random generator. [ 3.407993] random: ln: uninitialized urandom read (6 bytes read) [ 3.487650] random: systemd: uninitialized urandom read (16 bytes read) [ 3.490284] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.494867] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.498989] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.086279] device-mapper: uevent: version 1.0.3 [ 4.088563] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.383399] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.739127] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.832263] scsi host0: ata_piix [ 4.859243] scsi host1: ata_piix [ 4.861048] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.863584] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.879786] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.771091] random: crng init done [ 9.772741] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.954545] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.052208] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.328815] SELinux: Disabled at runtime. [ 11.387206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.395864] systemd[1]: Detected virtualization kvm. [ 11.397478] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.894621] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.898233] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.904484] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.909378] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.913859] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.922228] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.927529] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ 12.045865] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.323078] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.648633] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.720474] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.756236] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.773802] EDAC sbridge: Ver: 1.1.2 [ 13.884452] Key type dns_resolver registered [ 14.189873] NFS: Registering the id_resolver key type [ 14.192115] Key type id_resolver registered [ 14.193795] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg638-client login: [ 41.159148] libcfs: loading out-of-tree module taints kernel. [ 41.237819] Key type ._llcrypt registered [ 41.239274] Key type .llcrypt registered [ 41.624251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.633172] alg: No test for adler32 (adler32-zlib) [ 42.696463] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 43.031567] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 44.671239] Key type lgssc registered [ 45.464861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.173004] hrtimer: interrupt took 3996417 ns [ 171.724417] Lustre: Mounted lustre-client [ 177.330696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.065216] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing check_logdir /tmp/testlogs/ [ 197.618978] Lustre: lustre-OST0000-osc-ffff9e99036be000: disconnect after 23s idle [ 200.051700] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing yml_node [ 205.307320] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 207.755508] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 210.452056] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 212.129884] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Apr 3 07:46:03 EDT 2026 [ 229.837049] LustreError: 7913:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9e99036be000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 229.890487] Lustre: Unmounted lustre-client [ 256.001915] Key type lgssc unregistered [ 256.376501] LNet: 8358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.388294] LNetError: 8358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 256.407149] LNet: Removed LNI 192.168.206.38@tcp [ 257.336278] Key type .llcrypt unregistered [ 257.338958] Key type ._llcrypt unregistered [ 289.043212] Key type ._llcrypt registered [ 289.045344] Key type .llcrypt registered [ 289.255084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 289.277665] alg: No test for adler32 (adler32-zlib) [ 290.286379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 290.361495] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 290.382270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 295.399705] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 295.506612] LNet: 9230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.516416] LNetError: 9230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.566042] LNet: Removed LNI 192.168.206.38@tcp [ 296.385438] Key type .llcrypt unregistered [ 296.395723] Key type ._llcrypt unregistered [ 303.195962] Key type ._llcrypt registered [ 303.200469] Key type .llcrypt registered [ 305.224197] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:47:36 (1775216856) [ 305.663765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 305.705606] alg: No test for adler32 (adler32-zlib) [ 306.732497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.856740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.900988] LNet: 9648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.924125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.058102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 307.095813] LNet: 9655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 314.636355] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:47:46 (1775216866) [ 314.714745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.742074] LNet: 10225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.757323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.837602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 321.388325] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:47:53 (1775216873) [ 321.527173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.558023] LNet: 10813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.597143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.680629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 327.989295] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:47:59 (1775216879) [ 328.093301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.133447] LNet: 11395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.162196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.327018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 334.521151] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:48:06 (1775216886) [ 334.614436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.656175] LNet: 11980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.679822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.800510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 335.398154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 342.722112] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:48:14 (1775216894) [ 343.024606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.055699] LNet: 12578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.103208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.201197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 343.741239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 344.256480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 351.613303] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:48:23 (1775216903) [ 352.076623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.116134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.204683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 360.278768] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:48:32 (1775216912) [ 360.655294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.703350] LNet: 13759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.716361] LNet: 13759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 360.741339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.839751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 360.895642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.652919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 361.736783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 362.715423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 362.812676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 363.340440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 363.410566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 364.132298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 364.187359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 364.943506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 364.980771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 365.577602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 365.625911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 366.200767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 366.254662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 366.820442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 366.850676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 367.235986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 367.278686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 367.859109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 367.900887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 368.576631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 368.619480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 369.016518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 369.049332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 376.145595] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:48:47 (1775216927) [ 376.275570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.329676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.434205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 376.477034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 383.832944] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:48:55 (1775216935) [ 383.976462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.052981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.198775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 384.234236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 391.059119] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:49:02 (1775216942) [ 391.414131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.462503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.537171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 391.583250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 398.370399] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:49:10 (1775216950) [ 398.736173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.768684] LNet: 16203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.777501] LNet: 16203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 398.827557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.977752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 399.029852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 406.319366] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:49:18 (1775216958) [ 406.599992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.649480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.762097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 406.812360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 413.852242] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:49:25 (1775216965) [ 414.327468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.379914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.509430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 414.561694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 422.217703] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:49:33 (1775216973) [ 422.500451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.563940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.647893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.281238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.927494] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:49:43 (1775216983) [ 432.104753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.160811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.486697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.936350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.656075] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:49:53 (1775216993) [ 441.781351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.890710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.128542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.472589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.865871] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:50:01 (1775217001) [ 449.999847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.082831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.619622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.937517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.290868] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:50:10 (1775217010) [ 458.673916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.723546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.808289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.890212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.863737] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:50:17 (1775217017) [ 465.963327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.001570] LNet: 21031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.016524] LNet: 21031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 466.045327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.153618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.205142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.519264] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:50:25 (1775217025) [ 473.705815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.771156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.874857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 480.338374] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:50:32 (1775217032) [ 480.506702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.578482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.692629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 487.548503] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:50:39 (1775217039) [ 487.647391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.685178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.793748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 488.550639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 496.581323] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:50:48 (1775217048) [ 496.691755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.770935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.930151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 496.979183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 504.965479] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:50:56 (1775217056) [ 505.105208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.155934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.306145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 505.380265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 513.319497] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:51:04 (1775217064) [ 513.457387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.526696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.693894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 513.771410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 520.851810] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:51:12 (1775217072) [ 520.988655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.058270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.201787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.280031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 529.159712] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:51:21 (1775217081) [ 529.238697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.272766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.350929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 529.466996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 529.495345] LNetError: 25763:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 529.525634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 529.577343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 536.435663] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:51:28 (1775217088) [ 536.722067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.757414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.827493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.j1xbi0RLKj [ 536.888945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.j1xbi0RLKj [ 536.926467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.j1xbi0RLKj [ 545.094781] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:51:36 (1775217096) [ 545.757761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.848331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.950804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 546.022572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 546.126747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 554.600190] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:51:45 (1775217105) [ 554.675023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.733135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.884661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 558.946191] LNet: Message forwarding will be enabled in 2s [ 562.015147] LNet: Message forwarding enabled [ 562.517449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 562.548562] LNet: Message forwarding will stop in 2s to 6s [ 567.339483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 567.852997] LNet: Message forwarding disabled [ 569.519219] LNet: Message forwarding will be enabled in 2s [ 572.959506] LNet: Message forwarding enabled [ 578.046756] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:52:09 (1775217129) [ 578.180343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.705339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.777427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 586.346843] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:52:18 (1775217138) [ 586.771320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.860655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.919295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 586.980313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 587.018896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 587.071323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 587.130963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 587.196039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 587.244476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 587.290527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 587.328284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 587.389295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 587.425145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 587.465919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 587.504163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 587.568157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 587.644802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 587.712900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 587.780794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 587.836116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 587.904486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 587.957411] LNetError: 28835:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 595.160744] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:52:26 (1775217146) [ 595.469837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.492600] LNet: 29406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.499995] LNet: 29406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 595.533857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.609962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.493214] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:52:34 (1775217154) [ 604.114987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg638-server [ 606.818933] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:52:38 (1775217158) [ 606.981344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.069931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.136514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.195237] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 609.189726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[39-43]@tcp [ 609.231992] LNet: 30552:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 609.306255] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.39@tcp is being used as a gateway but routing feature is not turned on [ 609.331083] LNetError: 30339:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.39@tcp has gone from up to down [ 610.293990] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.39@tcp is being used as a gateway but routing feature is not turned on [ 610.310715] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 611.304835] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.41@tcp is being used as a gateway but routing feature is not turned on [ 611.319578] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 613.359424] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.39@tcp is being used as a gateway but routing feature is not turned on [ 613.393795] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 616.782816] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:52:48 (1775217168) [ 617.442743] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.43@tcp is being used as a gateway but routing feature is not turned on [ 617.456782] LNetError: 30339:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 619.001645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg638-server [ 621.227317] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:52:52 (1775217172) [ 621.435187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.456324] LNetError: 31487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.479128] LNet: Removed LNI 192.168.206.38@tcp [ 621.496172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.550141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.571197] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 622.195409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[39-44/2]@tcp [ 622.216514] LNet: 31604:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 622.224577] LNet: 31604:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 622.254686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[39-44/2]@tcp [ 622.271301] LNetError: 31491:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.39@tcp has gone from up to down [ 622.286960] LNetError: 31491:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 629.539788] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:53:01 (1775217181) [ 629.628591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.669823] LNetError: 32191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.684796] LNet: Removed LNI 192.168.206.38@tcp [ 629.732892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.843397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 629.908631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.947321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.023236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.126618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.209368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.317732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 630.473199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.541895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.626435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.709214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.744690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.880305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 631.017341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.089349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.226469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.341852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.396612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.473742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 631.580625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.653645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.765729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.857714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.909450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.982598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 632.065900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.115229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.201882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 632.293951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.343681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.424262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 632.505456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.545970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.640684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.085933] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:53:11 (1775217191) [ 642.241877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg638-server [ 643.767858] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:53:15 (1775217195) [ 645.431423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg638-server [ 647.205176] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:53:19 (1775217199) [ 648.348621] Key type .llcrypt unregistered [ 648.370112] Key type ._llcrypt unregistered [ 654.067262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 654.256190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 654.273163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 654.288567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 655.906598] Key type ._llcrypt registered [ 655.910990] Key type .llcrypt registered [ 656.116718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.132329] alg: No test for adler32 (adler32-zlib) [ 657.066410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.152452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.197434] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 657.796728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 657.820676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 659.698569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 659.742760] LNet: Removed LNI 10.1.2.3@tcp [ 659.877629] LNet: 34313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.893527] LNetError: 34313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.919463] LNet: Removed LNI 192.168.206.38@tcp [ 660.618531] Key type .llcrypt unregistered [ 660.624221] Key type ._llcrypt unregistered [ 671.767744] Key type ._llcrypt registered [ 671.771091] Key type .llcrypt registered [ 672.877963] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:53:44 (1775217224) [ 673.570915] Key type .llcrypt unregistered [ 673.575025] Key type ._llcrypt unregistered [ 678.850871] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.995307] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.999506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 679.005380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 681.496507] Key type ._llcrypt registered [ 681.498966] Key type .llcrypt registered [ 681.682050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.699444] alg: No test for adler32 (adler32-zlib) [ 682.628469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.687818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.728888] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 683.531356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.557473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.115609] LNet: 35801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.125271] LNetError: 35801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.131964] LNet: Removed LNI 192.168.206.38@tcp [ 686.149397] LNet: Removed LNI 10.1.2.3@tcp [ 686.833798] Key type .llcrypt unregistered [ 686.840950] Key type ._llcrypt unregistered [ 697.325831] Key type ._llcrypt registered [ 697.328590] Key type .llcrypt registered [ 698.771950] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:54:10 (1775217250) [ 699.397272] Key type .llcrypt unregistered [ 699.399132] Key type ._llcrypt unregistered [ 706.002726] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 706.308347] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 706.319417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 706.333432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 708.898752] Key type ._llcrypt registered [ 708.901223] Key type .llcrypt registered [ 709.087853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.105437] alg: No test for adler32 (adler32-zlib) [ 710.059564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.131533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 710.163619] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 712.072545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 712.111393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 714.181891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 714.198991] LNet: Removed LNI 10.1.2.3@tcp [ 714.211691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 714.246721] LNet: Removed LNI 10.1.2.31@tcp [ 714.261752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 714.289280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 714.956368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 714.976345] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 716.035305] LNet: 37724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.051849] LNetError: 37724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 716.065257] LNet: Removed LNI 10.1.2.3@tcp [ 716.096171] LNet: Removed LNI 10.1.2.31@tcp [ 716.999663] Key type .llcrypt unregistered [ 717.008077] Key type ._llcrypt unregistered [ 730.225336] Key type ._llcrypt registered [ 730.227564] Key type .llcrypt registered [ 731.459888] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:54:43 (1775217283) [ 732.172771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.196368] alg: No test for adler32 (adler32-zlib) [ 733.175299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.244531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.279439] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 734.307390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.333992] LNet: 38691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.345281] LNetError: 38691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.383517] LNet: Removed LNI 192.168.206.38@tcp [ 734.410173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.496943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.38@tcp [ 734.512908] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 735.634986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.38@tcp [ 735.676217] LNet: Removed LNI 192.168.206.38@tcp [ 735.761349] LNet: 38826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.769996] Key type .llcrypt unregistered [ 736.776124] Key type ._llcrypt unregistered [ 750.140394] Key type ._llcrypt registered [ 750.142023] Key type .llcrypt registered [ 751.784899] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:55:03 (1775217303) [ 753.455236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg638-server [ 755.591119] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:55:07 (1775217307) [ 756.588469] Key type .llcrypt unregistered [ 756.594993] Key type ._llcrypt unregistered [ 762.150098] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 762.444404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.454283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.466670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 765.169693] Key type ._llcrypt registered [ 765.174068] Key type .llcrypt registered [ 765.398984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 765.412696] alg: No test for adler32 (adler32-zlib) [ 766.291292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.355618] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 767.139777] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 769.590139] LNet: 40663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.598122] LNetError: 40663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.607373] LNet: Removed LNI 192.168.206.38@tcp1 [ 769.627124] LNet: Removed LNI 10.1.2.3@tcp2 [ 770.439491] Key type .llcrypt unregistered [ 770.443573] Key type ._llcrypt unregistered [ 781.613236] Key type ._llcrypt registered [ 781.615947] Key type .llcrypt registered [ 783.114757] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:55:34 (1775217334) [ 783.515555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.531973] alg: No test for adler32 (adler32-zlib) [ 784.582473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.699476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.736099] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 785.986425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.029357] LNet: 41632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.040342] LNetError: 41632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.069830] LNet: Removed LNI 192.168.206.38@tcp [ 786.118675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.217356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 786.270877] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 793.574421] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:55:45 (1775217345) [ 793.795304] LNet: 42223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.822319] LNetError: 42223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.848719] LNet: Removed LNI 192.168.206.38@tcp [ 794.511919] Key type .llcrypt unregistered [ 794.515271] Key type ._llcrypt unregistered [ 800.530068] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.748790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.756306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.763337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.485427] Key type ._llcrypt registered [ 803.488276] Key type .llcrypt registered [ 803.759795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.799873] alg: No test for adler32 (adler32-zlib) [ 804.820370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.934268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.976164] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 806.145977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.189867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.433438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.461579] LNet: 42967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.472230] LNetError: 42967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.481216] LNet: Removed LNI 192.168.206.38@tcp [ 808.497309] LNet: Removed LNI 10.1.2.3@tcp [ 808.535358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.611278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 808.653574] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 808.664649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.272990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.320430] LNet: 42994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.332518] LNetError: 42994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.348317] LNetError: 42994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 809.359156] LNet: Removed LNI 192.168.206.38@tcp [ 809.386623] LNet: Removed LNI 10.1.2.3@tcp [ 809.419381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.521704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 809.563616] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 809.571575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.064104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 810.093923] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 811.056794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.089767] LNet: 43156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.105209] LNetError: 43156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.110192] LNetError: 43156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 811.127075] LNet: Removed LNI 192.168.206.38@tcp [ 811.136949] LNet: Removed LNI 10.1.2.3@tcp [ 811.153529] LNet: Removed LNI 192.168.206.38@tcp2 [ 811.185868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.315981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 811.359737] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 811.368105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.376750] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 819.631905] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:56:11 (1775217371) [ 819.834330] LNet: 43755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.840441] LNetError: 43755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.852027] LNetError: 43755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 819.861112] LNet: Removed LNI 192.168.206.38@tcp [ 819.863933] LNet: Removed LNI 10.1.2.3@tcp [ 819.884249] LNet: Removed LNI 192.168.206.38@tcp2 [ 820.591817] Key type .llcrypt unregistered [ 820.594126] Key type ._llcrypt unregistered [ 827.514957] Key type ._llcrypt registered [ 827.518445] Key type .llcrypt registered [ 827.891337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.922975] alg: No test for adler32 (adler32-zlib) [ 829.003666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.075299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.103156] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 829.894188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.919713] LNet: 44150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.925388] LNetError: 44150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.945107] LNet: Removed LNI 192.168.206.38@tcp [ 829.974648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.047516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 830.076861] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 830.805789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.866765] LNet: 44180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.885733] LNetError: 44180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.905729] LNet: Removed LNI 192.168.206.38@tcp [ 830.975538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.108863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 831.156901] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 832.121976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.172115] LNet: 44210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.176429] LNetError: 44210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.200668] LNet: Removed LNI 192.168.206.38@tcp [ 832.252881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.384121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 832.422996] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 832.917061] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 833.019349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 833.022107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 833.040949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.458471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.477893] LNet: 44290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.483935] LNetError: 44290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.497760] LNet: Removed LNI 192.168.206.38@tcp [ 835.518310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.625278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 835.677974] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 842.227434] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:56:34 (1775217394) [ 842.522559] LNet: 44888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.528634] LNetError: 44888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.548463] LNet: Removed LNI 192.168.206.38@tcp [ 843.043112] Key type .llcrypt unregistered [ 843.044885] Key type ._llcrypt unregistered [ 849.277501] Key type ._llcrypt registered [ 849.279638] Key type .llcrypt registered [ 849.484770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.515120] alg: No test for adler32 (adler32-zlib) [ 850.529557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.624867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.664847] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 851.819121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.857588] LNet: 45286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.870415] LNetError: 45286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.905956] LNet: Removed LNI 192.168.206.38@tcp [ 851.940876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.002830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 852.025711] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 858.355744] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:56:50 (1775217410) [ 858.539038] LNet: 45873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.555596] LNetError: 45873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.572673] LNet: Removed LNI 192.168.206.38@tcp [ 859.347646] Key type .llcrypt unregistered [ 859.356352] Key type ._llcrypt unregistered [ 865.602977] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 865.732953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 865.745067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 865.759282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 867.328762] Key type ._llcrypt registered [ 867.332993] Key type .llcrypt registered [ 867.580942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.619598] alg: No test for adler32 (adler32-zlib) [ 868.604223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.676506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.712492] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 869.821774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.841722] LNet: 46315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.847284] LNetError: 46315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.872787] LNet: Removed LNI 192.168.206.38@tcp [ 869.904235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.024888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 870.082491] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 877.918423] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:57:09 (1775217429) [ 878.353861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.374495] LNet: 46913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.385528] LNetError: 46913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.401898] LNet: Removed LNI 192.168.206.38@tcp [ 878.439529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.543919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 884.909150] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:57:16 (1775217436) [ 885.233624] LNet: 47496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.754944] Key type .llcrypt unregistered [ 885.757602] Key type ._llcrypt unregistered [ 893.249551] Key type ._llcrypt registered [ 893.255158] Key type .llcrypt registered [ 893.514886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.541465] alg: No test for adler32 (adler32-zlib) [ 894.573288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.665875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.722754] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 895.802502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.851217] LNet: 47893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.862425] LNetError: 47893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.885761] LNet: Removed LNI 192.168.206.38@tcp [ 895.907603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.992540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 896.018752] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 903.322875] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:57:35 (1775217455) [ 903.486450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.541447] LNet: 48485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.547956] LNetError: 48485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.575397] LNet: Removed LNI 192.168.206.38@tcp [ 903.596938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.672987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 910.052886] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:57:41 (1775217461) [ 911.638992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 914.369719] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:57:45 (1775217465) [ 916.418360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 918.674958] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:57:50 (1775217470) [ 920.887547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 923.733289] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:57:54 (1775217474) [ 925.770338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 927.396927] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:57:59 (1775217479) [ 929.454459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 931.719452] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:58:03 (1775217483) [ 932.001604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.041297] LNet: 50542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.077876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.139628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.163597] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 933.079841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.38 [ 933.105821] LNet: 50675:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 933.122608] LNet: Removed LNI 192.168.206.38@tcp [ 939.846465] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:58:11 (1775217491) [ 939.990833] LNet: 51239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.001569] LNetError: 51239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.043223] LNet: Removed LNI 192.168.206.38@tcp [ 940.704907] Key type .llcrypt unregistered [ 940.707402] Key type ._llcrypt unregistered [ 947.382405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 947.619820] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 947.626744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 947.629417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 950.315486] Key type ._llcrypt registered [ 950.320250] Key type .llcrypt registered [ 950.484972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.512342] alg: No test for adler32 (adler32-zlib) [ 951.810456] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 958.047307] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:58:30 (1775217510) [ 958.428962] LNet: 52151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.439121] LNetError: 52151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.463592] LNet: Removed LNI 10.1.2.3@tcp [ 959.233451] Key type .llcrypt unregistered [ 959.235168] Key type ._llcrypt unregistered [ 965.357685] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.637542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.643277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.645988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.369545] Key type ._llcrypt registered [ 968.372159] Key type .llcrypt registered [ 968.531210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.552355] alg: No test for adler32 (adler32-zlib) [ 969.599154] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.025928] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:58:48 (1775217528) [ 977.260426] LNet: 53044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.281962] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.333929] LNet: Removed LNI 10.1.2.3@tcp [ 978.189705] Key type .llcrypt unregistered [ 978.199377] Key type ._llcrypt unregistered [ 985.735722] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.050237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.086432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.101088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.916593] Key type ._llcrypt registered [ 988.922381] Key type .llcrypt registered [ 989.136473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.167937] alg: No test for adler32 (adler32-zlib) [ 990.194364] LNet: 53346:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 990.232992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.053117] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:59:09 (1775217549) [ 998.188448] LNet: 53930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.205884] LNetError: 53930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.242024] LNet: Removed LNI 10.1.2.3@tcp [ 999.140763] Key type .llcrypt unregistered [ 999.148423] Key type ._llcrypt unregistered [ 1005.835823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.144552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.148174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.156446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.154989] Key type ._llcrypt registered [ 1009.161256] Key type .llcrypt registered [ 1009.451456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.478727] alg: No test for adler32 (adler32-zlib) [ 1010.358408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.437465] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1019.652980] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:59:31 (1775217571) [ 1020.190893] LNet: 55167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.196024] LNetError: 55167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.214084] LNet: Removed LNI 10.1.2.3@tcp [ 1020.888768] Key type .llcrypt unregistered [ 1020.891511] Key type ._llcrypt unregistered [ 1028.352495] Key type ._llcrypt registered [ 1028.355698] Key type .llcrypt registered [ 1028.570076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.595237] alg: No test for adler32 (adler32-zlib) [ 1029.557841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1029.620461] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1038.476581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1039.554125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_interrupt -r 1 [ 1039.646711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_interrupt -r 1 [ 1039.709856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1039.750167] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1039.905722] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1040.145963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.198862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1040.249479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.321712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.367230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.438560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.514762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.554777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.588399] LNet: 55607:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.614626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.668846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.724308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.763305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.821853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.881796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.908967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1041.452956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_dropped -r 1 [ 1041.507614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_dropped -r 1 [ 1041.575179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1041.595268] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1041.943707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.015977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.096933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.163530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.232485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.280776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.327569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.376909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.411578] LNet: 55716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.450650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.487510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.544324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.597665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.648984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.689719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.723869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.427301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_aborted -r 1 [ 1043.485364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_aborted -r 1 [ 1043.550069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1043.597397] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1043.996450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.071076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1044.129414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.209531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.238710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.267891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.315414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.355928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.377671] LNet: 55826:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.403831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.448893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.497194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.535195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.596971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.653565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.708477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.369370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_no_route -r 1 [ 1045.403261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_no_route -r 1 [ 1045.437308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1045.644859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.682961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.716164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.756756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.805763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.848245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.898273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.938257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.975557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.020178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.076972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.103330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.144574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.183644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.211888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1046.653604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_timeout -r 1 [ 1046.679455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_timeout -r 1 [ 1046.710885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1046.724705] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1046.727048] LNet: Skipped 2 previous similar messages [ 1046.962618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.010414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1047.043927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.074758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.109737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.139968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.193924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.242452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.265856] LNet: 56044:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.275978] LNet: 56044:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1047.327765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.377624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.441096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.500541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.561860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.617211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.705290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1048.306963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_error -r 1 [ 1048.355761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_error -r 1 [ 1048.417119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1048.683302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.736795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.779856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.821927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.863687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.899972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.936825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.991252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.036885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.065540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.089306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.114680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.158711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.200708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.249715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.584340] LNet: 56195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.594750] LNetError: 56195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.614341] LNet: Removed LNI 192.168.206.38@tcp [ 1050.152850] Key type .llcrypt unregistered [ 1050.156503] Key type ._llcrypt unregistered [ 1069.259239] Key type ._llcrypt registered [ 1069.261577] Key type .llcrypt registered [ 1070.808992] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:00:22 (1775217622) [ 1071.746305] Key type .llcrypt unregistered [ 1071.748095] Key type ._llcrypt unregistered [ 1078.500598] Key type ._llcrypt registered [ 1078.504505] Key type .llcrypt registered [ 1078.803406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.835341] alg: No test for adler32 (adler32-zlib) [ 1079.828565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1079.873932] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1088.408402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1100.009200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.064765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1100.093347] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1101.614315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_interrupt -r 1 [ 1101.649620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_interrupt -r 1 [ 1101.692317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_interrupt -r 1 [ 1101.726831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_interrupt -r 1 [ 1101.761731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_interrupt -r 1 [ 1101.807840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_interrupt -r 1 [ 1101.844358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_interrupt -r 1 [ 1101.887094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_interrupt -r 1 [ 1101.946846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1101.983395] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1101.998807] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1102.206744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.235462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.281939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.332501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.378499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.421605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.475375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.527527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.553239] LNet: 57663:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.588614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.644793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.696733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.752855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.807653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.855845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.902815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.361469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_dropped -r 1 [ 1103.391223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_dropped -r 1 [ 1103.429247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_dropped -r 1 [ 1103.467867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_dropped -r 1 [ 1103.502076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_dropped -r 1 [ 1103.535707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_dropped -r 1 [ 1103.590634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_dropped -r 1 [ 1103.651948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_dropped -r 1 [ 1103.730507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1103.765469] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1103.771597] LNet: Skipped 4 previous similar messages [ 1104.035804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.097215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.150717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.191245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.233979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.269700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.309948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.411753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.431603] LNet: 57784:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.467246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.527380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.576664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.624413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.651349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.685102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.742752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.253338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_aborted -r 1 [ 1105.288380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_aborted -r 1 [ 1105.335640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_aborted -r 1 [ 1105.394190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_aborted -r 1 [ 1105.472929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_aborted -r 1 [ 1105.536837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_aborted -r 1 [ 1105.590594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_aborted -r 1 [ 1105.635311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_aborted -r 1 [ 1105.666631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1105.688573] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1105.701267] LNet: Skipped 2 previous similar messages [ 1105.947691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.981651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1106.020468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.050451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.079499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.112538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.151413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.182835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.215405] LNet: 57905:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.239409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.269418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.295528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.326458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.374206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.415545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.444788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.975916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_no_route -r 1 [ 1107.023806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_no_route -r 1 [ 1107.076629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_no_route -r 1 [ 1107.136365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_no_route -r 1 [ 1107.185803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_no_route -r 1 [ 1107.240538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_no_route -r 1 [ 1107.303918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_no_route -r 1 [ 1107.369654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_no_route -r 1 [ 1107.440240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1107.872443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.922936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.994315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.045907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.119537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.211045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.274169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.352384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.403622] LNet: 58039:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.454917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.487343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.510165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.537450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.586249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.646224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.693795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.317267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_timeout -r 1 [ 1109.377993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_timeout -r 1 [ 1109.427293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_timeout -r 1 [ 1109.480889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_timeout -r 1 [ 1109.539696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_timeout -r 1 [ 1109.596861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_timeout -r 1 [ 1109.639932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_timeout -r 1 [ 1109.690755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_timeout -r 1 [ 1109.756844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1109.778805] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1109.785907] LNet: Skipped 5 previous similar messages [ 1110.038888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.088741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.138528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.176343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.211337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.246656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.308824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.364070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.412783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.469262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.522520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.578453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.623182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.666595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.695329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.270291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_error -r 1 [ 1111.307782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_error -r 1 [ 1111.335464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_error -r 1 [ 1111.363670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_error -r 1 [ 1111.390316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_error -r 1 [ 1111.429217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_error -r 1 [ 1111.462821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_error -r 1 [ 1111.495533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_error -r 1 [ 1111.524239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1111.808950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.846383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.891603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.939725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.016567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.052247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.083613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.119730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.147775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.177500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.203515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.243300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.274455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.308506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.347286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.562390] LNet: 58358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.579497] LNetError: 58358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.603265] LNet: Removed LNI 192.168.206.38@tcp [ 1119.653979] LNet: Removed LNI 192.168.206.38@tcp1 [ 1120.212806] Key type .llcrypt unregistered [ 1120.215435] Key type ._llcrypt unregistered [ 1139.061896] Key type ._llcrypt registered [ 1139.065373] Key type .llcrypt registered [ 1140.168218] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:01:32 (1775217692) [ 1141.380355] Key type .llcrypt unregistered [ 1141.382875] Key type ._llcrypt unregistered [ 1147.581354] Key type ._llcrypt registered [ 1147.583854] Key type .llcrypt registered [ 1147.779690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.796771] alg: No test for adler32 (adler32-zlib) [ 1148.730458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1148.781090] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1156.757835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1157.762630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_dropped -r 1 [ 1157.801349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_dropped -r 1 [ 1157.847230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1157.884037] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1158.178136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.235351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1158.289588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.341660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.406479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.447267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.495550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.536638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.555038] LNet: 59609:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.573389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.608213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.646731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.683563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.715902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.754285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.812777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1159.538756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_error -r 1 [ 1159.577616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_error -r 1 [ 1159.619967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1159.656716] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1159.970535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.047602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1160.108672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.157659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.208677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.262193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.310951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.381413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.406794] LNet: 59719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.429606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.460995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.514273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.547133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.588830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.642706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.698847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.328126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_timeout -r 1 [ 1161.380161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_timeout -r 1 [ 1161.449477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1161.481484] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1161.757075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.799325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.837778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.885543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.968640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.002802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.041313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.112050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.147306] LNet: 59829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.169923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.219762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.255951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.290862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.343887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.403644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.431449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.719061] LNet: 59870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.724090] LNetError: 59870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.756112] LNet: Removed LNI 192.168.206.38@tcp [ 1163.234223] Key type .llcrypt unregistered [ 1163.237237] Key type ._llcrypt unregistered [ 1181.784783] Key type ._llcrypt registered [ 1181.787676] Key type .llcrypt registered [ 1183.067046] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:02:14 (1775217734) [ 1183.817514] Key type .llcrypt unregistered [ 1183.823226] Key type ._llcrypt unregistered [ 1190.911404] Key type ._llcrypt registered [ 1190.913260] Key type .llcrypt registered [ 1191.255107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.295355] alg: No test for adler32 (adler32-zlib) [ 1192.304652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.366417] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1200.833311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1210.842671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.882185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1210.915553] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1212.226830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_dropped -r 1 [ 1212.263669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e remote_dropped -r 1 [ 1212.321490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_dropped -r 1 [ 1212.380579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e remote_dropped -r 1 [ 1212.459682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e remote_dropped -r 1 [ 1212.533314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e remote_dropped -r 1 [ 1212.576063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e remote_dropped -r 1 [ 1212.640442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e remote_dropped -r 1 [ 1212.703254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1212.737975] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1212.755740] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.138@tcp [ 1213.039577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.069749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.096669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.129321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.153707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.180788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.206303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.227592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.250115] LNet: 61335:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.268253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.294806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.328973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.370449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.428976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.486967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.520371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.562208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.081365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_error -r 1 [ 1214.107216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e remote_error -r 1 [ 1214.141175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_error -r 1 [ 1214.184887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e remote_error -r 1 [ 1214.227292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e remote_error -r 1 [ 1214.274631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e remote_error -r 1 [ 1214.316380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e remote_error -r 1 [ 1214.358754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e remote_error -r 1 [ 1214.398492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1214.427483] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1214.435330] LNet: Skipped 2 previous similar messages [ 1214.662715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.689367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.729646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.766152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.796277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.838658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.894344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.938102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.970488] LNet: 61458:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.992319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.033422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.084536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.124244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.199460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.268403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.321133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1215.383790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.970406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_timeout -r 1 [ 1216.007336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e remote_timeout -r 1 [ 1216.048289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_timeout -r 1 [ 1216.092833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e remote_timeout -r 1 [ 1216.139859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e remote_timeout -r 1 [ 1216.176332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e remote_timeout -r 1 [ 1216.208215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e remote_timeout -r 1 [ 1216.239883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e remote_timeout -r 1 [ 1216.282977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.138@tcp [ 1216.302900] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1216.566767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.612183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.644927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.680336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.715648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.762580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.825748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.881547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.908674] LNet: 61582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.931542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.970727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.010458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.045634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.094218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.147151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.188426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.240670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.095902] LNet: 61660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.101629] LNetError: 61660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.115859] LNet: Removed LNI 192.168.206.38@tcp [ 1222.128712] LNet: Removed LNI 192.168.206.38@tcp1 [ 1222.739776] Key type .llcrypt unregistered [ 1222.749666] Key type ._llcrypt unregistered [ 1240.463979] Key type ._llcrypt registered [ 1240.470450] Key type .llcrypt registered [ 1241.780941] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:03:13 (1775217793) [ 1242.644679] Key type .llcrypt unregistered [ 1242.649166] Key type ._llcrypt unregistered [ 1249.980681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1250.250441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1250.256734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1250.270511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1253.215706] Key type ._llcrypt registered [ 1253.218643] Key type .llcrypt registered [ 1253.525886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.564637] alg: No test for adler32 (adler32-zlib) [ 1254.580480] LNet: 62737:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1254.605190] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1254.674736] LNet: 62751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.686402] LNetError: 62751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.745952] LNet: Removed LNI 192.168.206.38@tcp [ 1255.756449] Key type .llcrypt unregistered [ 1255.758300] Key type ._llcrypt unregistered [ 1263.398243] Key type ._llcrypt registered [ 1263.400353] Key type .llcrypt registered [ 1263.638347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1263.657320] alg: No test for adler32 (adler32-zlib) [ 1264.727046] LNet: 63010:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1264.750324] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1264.762600] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1264.833787] LNet: 63024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.844149] LNetError: 63024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.854593] LNet: Removed LNI 192.168.206.38@tcp [ 1264.866255] LNet: Removed LNI 10.1.2.3@tcp1 [ 1265.843662] Key type .llcrypt unregistered [ 1265.846869] Key type ._llcrypt unregistered [ 1273.212256] Key type ._llcrypt registered [ 1273.220972] Key type .llcrypt registered [ 1273.510525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1273.547813] alg: No test for adler32 (adler32-zlib) [ 1274.544262] LNet: 63283:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1274.555080] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1274.557584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1274.599605] LNet: 63297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.609915] LNetError: 63297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.620426] LNet: Removed LNI 192.168.206.38@tcp [ 1274.640522] LNet: Removed LNI 10.1.2.3@tcp [ 1275.318423] Key type .llcrypt unregistered [ 1275.322090] Key type ._llcrypt unregistered [ 1282.856476] Key type ._llcrypt registered [ 1282.864793] Key type .llcrypt registered [ 1283.048597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.072366] alg: No test for adler32 (adler32-zlib) [ 1284.033477] LNet: 63558:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1284.067561] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1284.081079] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1284.147297] LNet: 63572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.160298] LNetError: 63572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.175782] LNet: Removed LNI 192.168.206.38@tcp [ 1284.191788] LNet: Removed LNI 10.1.2.3@tcp [ 1285.091766] Key type .llcrypt unregistered [ 1285.096196] Key type ._llcrypt unregistered [ 1293.796370] Key type ._llcrypt registered [ 1293.800290] Key type .llcrypt registered [ 1294.050896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.088024] alg: No test for adler32 (adler32-zlib) [ 1295.102630] LNet: 63836:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1295.139551] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1295.148862] LNetError: 63836:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1295.163420] LNetError: Error -100 starting up LNI tcp [ 1295.177689] LNet: Removed LNI 192.168.206.38@tcp [ 1296.021047] Key type .llcrypt unregistered [ 1296.031910] Key type ._llcrypt unregistered [ 1310.172541] Key type ._llcrypt registered [ 1310.178426] Key type .llcrypt registered [ 1311.808851] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:04:23 (1775217863) [ 1312.787498] Key type .llcrypt unregistered [ 1312.792845] Key type ._llcrypt unregistered [ 1318.177502] Key type ._llcrypt registered [ 1318.182144] Key type .llcrypt registered [ 1318.421411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.437914] alg: No test for adler32 (adler32-zlib) [ 1319.383725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1319.402559] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1327.447767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1328.252499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e network_timeout -r 1 [ 1328.299380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e network_timeout -r 1 [ 1328.329354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1328.360122] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1328.587417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.607372] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1328.662689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1328.706582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.770676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.830990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.907351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.983334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.031902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.079031] LNet: 65092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.134867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.183302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.308895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.356322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.419113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.490916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.101605] LNet: 65131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.117883] LNetError: 65131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.147529] LNet: Removed LNI 192.168.206.38@tcp [ 1330.721673] Key type .llcrypt unregistered [ 1330.726839] Key type ._llcrypt unregistered [ 1354.070906] Key type ._llcrypt registered [ 1354.076773] Key type .llcrypt registered [ 1354.354621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.398968] alg: No test for adler32 (adler32-zlib) [ 1355.294209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.317048] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1363.659647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1372.436922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.498556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1372.544757] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1374.021837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e network_timeout -r 1 [ 1374.072183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e network_timeout -r 1 [ 1374.125770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e network_timeout -r 1 [ 1374.182954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e network_timeout -r 1 [ 1374.244445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e network_timeout -r 1 [ 1374.294527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e network_timeout -r 1 [ 1374.346606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e network_timeout -r 1 [ 1374.402502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e network_timeout -r 1 [ 1374.439907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1374.458393] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1374.666410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1374.719394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1374.754310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1374.783679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1374.816923] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1374.819762] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.138@tcp [ 1374.830108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1374.882302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1374.914601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1374.952410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1374.982970] LNet: 66012:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1375.006176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1375.044539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1375.087977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.133986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1375.198392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.244923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.888695] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1380.152049] LNet: 66086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.156534] LNetError: 66086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.162559] LNet: Removed LNI 192.168.206.38@tcp [ 1380.172373] LNet: Removed LNI 192.168.206.38@tcp1 [ 1380.539524] Key type .llcrypt unregistered [ 1380.541208] Key type ._llcrypt unregistered [ 1399.742431] Key type ._llcrypt registered [ 1399.745785] Key type .llcrypt registered [ 1401.074375] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:05:52 (1775217952) [ 1401.822994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.849506] alg: No test for adler32 (adler32-zlib) [ 1402.929934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.027671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1403.055600] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1404.044141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1404.071305] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1405.009893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 1405.060295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1405.103486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1405.131674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1405.168752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1405.211841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1405.254621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.38@tcp [ 1405.278914] LNetError: 67185:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.38@tcp to recovery [ 1406.047307] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1406.047361] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1408.095833] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1408.860769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1412.192206] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1412.358051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1412.384775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1412.438677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.452767] LNet: 67293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.457815] LNetError: 67293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.471257] LNet: Removed LNI 192.168.206.38@tcp [ 1413.483372] LNetError: 67293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.517435] LNet: Removed LNI 192.168.206.38@tcp1 [ 1413.581442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.732154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.779360] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1415.009177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1415.063983] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1416.321599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 1416.437061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1416.538172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1416.606223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1416.670200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1416.713168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1416.790384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1416.853035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.38@tcp [ 1416.873785] LNetError: 67565:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.38@tcp to recovery [ 1417.767513] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1420.520430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1423.905944] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1423.921929] LNet: Skipped 1 previous similar message [ 1428.487894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1428.536994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.566573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1436.151793] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:06:27 (1775217987) [ 1436.269974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.308092] LNet: 68277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.325985] LNetError: 68277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.343413] LNet: Removed LNI 192.168.206.38@tcp [ 1437.359759] LNet: Removed LNI 192.168.206.38@tcp1 [ 1437.407415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.485381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.524301] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1438.254673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1438.275596] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1438.985352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 1439.090481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1439.138594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1439.188756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1439.224624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1439.268646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1439.315148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.38@tcp --health 0 [ 1439.439541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1439.520399] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.38@tcp [ 1440.543772] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1440.549109] LNet: Skipped 1 previous similar message [ 1454.964887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1455.235798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1455.290866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.351391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.38@tcp --health 500 [ 1455.519442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1461.227899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1461.308448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.334803] LNet: 68766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.342923] LNetError: 68766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.351653] LNetError: 68766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1461.358464] LNet: Removed LNI 192.168.206.38@tcp [ 1461.370632] LNet: Removed LNI 192.168.206.38@tcp1 [ 1461.393567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.477855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1461.519734] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1462.542516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1462.591072] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1463.556509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 1463.608747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.666382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1463.701715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1463.757669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1463.803806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1463.840586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1463.875861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.38@tcp --health 0 [ 1464.543469] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1464.549941] LNet: Skipped 3 previous similar messages [ 1465.257211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1475.485463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1475.527394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1475.564893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.485447] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:07:14 (1775218034) [ 1484.085261] LNet: 69814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.103176] LNetError: 69814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.110406] LNetError: 69814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1484.115624] LNet: Removed LNI 192.168.206.38@tcp [ 1484.127437] LNet: Removed LNI 192.168.206.38@tcp1 [ 1484.804614] Key type .llcrypt unregistered [ 1484.807367] Key type ._llcrypt unregistered [ 1491.145429] Key type ._llcrypt registered [ 1491.147031] Key type .llcrypt registered [ 1491.231476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1491.252399] alg: No test for adler32 (adler32-zlib) [ 1492.131392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1492.171681] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1501.307325] Lustre: DEBUG MARKER: Initial discovery [ 1501.489707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.138@tcp [ 1504.009476] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1504.125381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1504.185328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.138@tcp [ 1504.230711] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1504.283374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1504.373107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.138@tcp -r 1 -m GET -l 3 [ 1504.435261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.138@tcp -r 1 -m GET -e local_error [ 1504.488685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.138@tcp -r 1 -m PUT -l 6 [ 1504.493268] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1506.147814] Lustre: DEBUG MARKER: Force oleg638-client.virtnet to discover 192.168.206.138@tcp (in background) [ 1506.272767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.138@tcp [ 1508.132962] Lustre: DEBUG MARKER: Force oleg638-server to discover 192.168.206.38@tcp [ 1509.416687] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1510.871991] Lustre: DEBUG MARKER: Wait for 70570 [ 1512.655628] Lustre: DEBUG MARKER: Finished wait on 70570 [ 1512.736305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1512.813779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.835997] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1513.120924] LNet: 71028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.132985] LNetError: 71028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.149510] LNet: Removed LNI 192.168.206.38@tcp [ 1513.606542] Key type .llcrypt unregistered [ 1513.609585] Key type ._llcrypt unregistered [ 1531.279859] Key type ._llcrypt registered [ 1531.286481] Key type .llcrypt registered [ 1532.586572] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:08:04 (1775218084) [ 1533.288862] Key type .llcrypt unregistered [ 1533.305399] Key type ._llcrypt unregistered [ 1539.849615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1540.026610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1540.030220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1540.043983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1542.585254] Key type ._llcrypt registered [ 1542.588527] Key type .llcrypt registered [ 1542.777967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.804621] alg: No test for adler32 (adler32-zlib) [ 1543.705690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.766584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.790656] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1544.601976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1544.619778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1553.136914] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:08:24 (1775218104) [ 1553.228438] LNet: 73100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.237546] LNetError: 73100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.246409] LNet: Removed LNI 192.168.206.38@tcp [ 1553.268280] LNet: Removed LNI 10.1.2.3@tcp [ 1553.923540] Key type .llcrypt unregistered [ 1553.929465] Key type ._llcrypt unregistered [ 1561.262172] Key type ._llcrypt registered [ 1561.264735] Key type .llcrypt registered [ 1561.569308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1561.601847] alg: No test for adler32 (adler32-zlib) [ 1562.601509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.668873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.686842] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1563.399730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.439060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1564.380378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.38@tcp [ 1565.407576] LNetError: Unexpected error -22 connecting to 192.168.206.38@tcp at host 192.168.206.38:988 [ 1570.379610] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:08:42 (1775218122) [ 1570.524927] LNet: 74236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.534744] LNetError: 74236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.549276] LNet: Removed LNI 192.168.206.38@tcp [ 1570.561458] LNet: Removed LNI 10.1.2.3@tcp [ 1571.384369] Key type .llcrypt unregistered [ 1571.386183] Key type ._llcrypt unregistered [ 1577.236567] Key type ._llcrypt registered [ 1577.250137] Key type .llcrypt registered [ 1577.419177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.447709] alg: No test for adler32 (adler32-zlib) [ 1578.388587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.456364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1578.493894] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1579.101098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1579.122285] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 1580.072987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.38@tcp1 --nid 192.168.206.38@tcp2 [ 1583.618621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp1 192.168.206.38@tcp2 [ 1583.640474] LNetError: 75074:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.38@tcp1 to 192.168.206.38@tcp2 - Route table may be misconfigured [ 1583.676340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp2 192.168.206.38@tcp1 [ 1589.472527] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:09:01 (1775218141) [ 1589.847493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.894717] LNet: 75647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.903927] LNetError: 75647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.908370] LNet: Removed LNI 192.168.206.38@tcp1 [ 1590.949939] LNet: Removed LNI 192.168.206.38@tcp2 [ 1590.996397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.059579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.081728] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1591.715814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1591.737350] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1592.539817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 1592.583878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.38@tcp [ 1592.632077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp1 192.168.206.38@tcp1 [ 1592.689789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.38@tcp -d 192.168.206.38@tcp -e network_timeout [ 1592.752535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -e network_timeout [ 1592.781503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -e network_timeout [ 1592.807617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -e network_timeout [ 1592.848291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1592.871651] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1592.903506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1593.001959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1593.056347] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1593.106296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1599.272635] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:09:11 (1775218151) [ 1599.658443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.694086] LNet: 76509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.708275] LNetError: 76509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.712679] LNetError: 76509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1599.721675] LNet: Removed LNI 192.168.206.38@tcp [ 1600.741543] LNet: Removed LNI 192.168.206.38@tcp1 [ 1600.756550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.827354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1601.150455] LNet: 76546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.486860] Key type .llcrypt unregistered [ 1601.491271] Key type ._llcrypt unregistered [ 1611.758288] Key type ._llcrypt registered [ 1611.760846] Key type .llcrypt registered [ 1613.119337] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:09:24 (1775218164) [ 1613.476978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1613.492915] alg: No test for adler32 (adler32-zlib) [ 1614.436621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.872647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1617.287734] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:09:29 (1775218169) [ 1617.634734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.659503] LNet: 77643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.693781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.751831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.795000] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1618.623876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.644220] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1619.398770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1619.449306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp1 [ 1619.498758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp1 [ 1625.746162] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:09:37 (1775218177) [ 1627.295058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg638-server [ 1629.060616] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:09:41 (1775218181) [ 1630.961951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg638-server [ 1633.227186] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:09:44 (1775218184) [ 1635.088653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg638-server [ 1637.157876] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:09:48 (1775218188) [ 1638.851370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg638-server [ 1640.816603] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:09:52 (1775218192) [ 1642.769794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg638-server [ 1644.939659] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:09:56 (1775218196) [ 1647.133982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg638-server [ 1649.415468] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:10:00 (1775218200) [ 1651.989379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg638-server [ 1654.037198] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:10:05 (1775218205) [ 1656.339779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg638-server [ 1658.027197] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:10:09 (1775218209) [ 1660.643192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg638-server [ 1662.878191] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:10:14 (1775218214) [ 1663.228685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.251668] LNet: 81521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.256739] LNetError: 81521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.264736] LNet: Removed LNI 192.168.206.38@tcp [ 1664.293568] LNet: Removed LNI 192.168.206.38@tcp1 [ 1664.349591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.442253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.475424] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1665.493391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1665.630309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1665.936883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.971442] LNet: 81669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.987450] LNetError: 81669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.994528] LNetError: 81669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1666.015758] LNet: Removed LNI 192.168.206.38@tcp [ 1666.040039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.110231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1666.156475] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1667.465740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1667.749565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.769887] LNetError: 81814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.783448] LNet: Removed LNI 192.168.206.38@tcp [ 1667.805636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.858593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.908066] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1669.035459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1669.160960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1669.479875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.534956] LNet: 81961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.546846] LNet: 81961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1670.580484] LNet: Removed LNI 192.168.206.38@tcp [ 1670.619448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.685756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1670.708232] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1671.227762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1671.541660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.566581] LNetError: 82106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.571492] LNetError: 82106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1671.583056] LNet: Removed LNI 192.168.206.38@tcp [ 1671.608268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.645605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.681057] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1672.669063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1672.763784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1672.954684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.033945] LNet: Removed LNI 192.168.206.38@tcp [ 1674.109339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.227730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1674.258606] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1675.425666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1675.855693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.879342] LNet: 82399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.886075] LNet: 82399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1675.893252] LNetError: 82399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.897918] LNetError: 82399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1676.969047] LNet: Removed LNI 192.168.206.38@tcp [ 1677.044226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.203396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.256123] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1678.847493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1678.913519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1679.195098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.230550] LNet: Removed LNI 192.168.206.38@tcp [ 1680.279183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.473315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1680.553069] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1682.143987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1682.588488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.688438] LNet: Removed LNI 192.168.206.38@tcp [ 1683.728918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.848064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.891244] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1684.937507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1685.066735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1688.002820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.034923] LNet: 82859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.043077] LNet: 82859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1688.050879] LNetError: 82859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.057827] LNetError: 82859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1688.075789] LNet: Removed LNI 192.168.206.38@tcp [ 1688.116171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.213806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1688.274330] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1689.569303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1689.966306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.046594] LNet: Removed LNI 192.168.206.38@tcp [ 1691.078250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.173904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.238699] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1692.336183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1692.425425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1692.751236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.804479] LNet: Removed LNI 192.168.206.38@tcp [ 1693.874220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.964985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1694.011291] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1694.791647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 1695.201985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.303434] LNet: Removed LNI 192.168.206.38@tcp [ 1696.362995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.446656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.472293] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1697.325540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1697.389779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.449582] LNet: Removed LNI 192.168.206.38@tcp [ 1697.469751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.536615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.574633] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1698.227915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1706.144022] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:10:57 (1775218257) [ 1706.260218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.303127] LNet: 84127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.309869] LNet: 84127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1706.319448] LNetError: 84127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.324663] LNetError: 84127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1706.348787] LNet: Removed LNI 192.168.206.38@tcp [ 1706.389110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.536111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1706.585598] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/180] [ 1708.644247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.686164] LNet: Removed LNI 192.168.206.38@tcp231 [ 1708.716194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.812702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1708.863545] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/180] [ 1708.879251] LNet: 84281:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1709.709553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1709.752400] LNet: Removed LNI 192.168.206.38@tcp231 [ 1709.782224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1709.823524] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/180] [ 1711.492154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.553969] LNet: Removed LNI 192.168.206.38@tcp231 [ 1711.593673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.698969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1711.730569] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/0] [ 1711.736286] LNet: 84433:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1712.158280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1712.220055] LNet: Removed LNI 192.168.206.38@tcp231 [ 1712.244501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1712.287965] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/0] [ 1713.647879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.684311] LNet: Removed LNI 192.168.206.38@tcp231 [ 1713.700753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.790711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1713.832907] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/60] [ 1713.842423] LNet: 84585:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.368240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1714.414965] LNet: Removed LNI 192.168.206.38@tcp231 [ 1714.432404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1714.457423] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/60] [ 1715.587446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.635681] LNet: Removed LNI 192.168.206.38@tcp231 [ 1715.664485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.718993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1715.739430] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/180] [ 1716.179662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1716.230713] LNet: Removed LNI 192.168.206.38@tcp231 [ 1716.261480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1716.294986] LNet: Added LNI 192.168.206.38@tcp231 [8/256/0/180] [ 1726.319638] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:11:17 (1775218277) [ 1726.449655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.513189] LNet: Removed LNI 192.168.206.38@tcp231 [ 1726.587743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.684924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1726.712827] LNet: Added LNI 192.168.206.38@tcp232 [8/256/0/180] [ 1727.513799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1727.606969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1727.657941] LNet: Removed LNI 192.168.206.38@tcp232 [ 1727.682140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1727.729835] LNet: Added LNI 192.168.206.38@tcp232 [8/256/0/180] [ 1728.414328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1728.468954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1728.500116] LNet: Removed LNI 192.168.206.38@tcp232 [ 1728.525840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1728.571208] LNet: Added LNI 192.168.206.38@tcp232 [8/256/0/180] [ 1729.353740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1729.418505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1729.487761] LNet: Removed LNI 192.168.206.38@tcp232 [ 1729.527051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1729.585465] LNet: Added LNI 192.168.206.38@tcp232 [8/256/0/180] [ 1730.781259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1730.834352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1730.916258] LNet: Removed LNI 192.168.206.38@tcp232 [ 1730.950971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1739.571819] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:11:30 (1775218290) [ 1740.037458] LNet: 86576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.048624] LNet: 86576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1740.748780] Key type .llcrypt unregistered [ 1740.752590] Key type ._llcrypt unregistered [ 1747.460301] Key type ._llcrypt registered [ 1747.465493] Key type .llcrypt registered [ 1747.852820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.874073] alg: No test for adler32 (adler32-zlib) [ 1748.773506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1748.803414] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1756.622663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1766.205225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.235901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1766.272138] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1767.934709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_interrupt -r 2 [ 1768.006304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_interrupt -r 2 [ 1768.043724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_interrupt -r 2 [ 1768.075211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_interrupt -r 2 [ 1768.109399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_interrupt -r 2 [ 1768.142779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_interrupt -r 2 [ 1768.192518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_interrupt -r 2 [ 1768.227664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_interrupt -r 2 [ 1768.297595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1768.323670] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1768.329576] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1768.798447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.835902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1768.902737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.964392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.028355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.106743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.164306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.260845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.309208] LNet: 87239:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.357512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.433785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.513641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.581736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.629662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.687724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1770.612636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_dropped -r 2 [ 1770.694032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_dropped -r 2 [ 1770.756598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_dropped -r 2 [ 1770.817579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_dropped -r 2 [ 1770.911799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_dropped -r 2 [ 1770.963372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_dropped -r 2 [ 1770.999985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_dropped -r 2 [ 1771.025173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_dropped -r 2 [ 1771.048455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1771.069348] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1771.266444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1771.291163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1771.315960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1771.351485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1771.399058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1771.468625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1771.510988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1771.550572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1771.575930] LNet: 87360:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1771.593582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1771.615908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1771.652884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1771.696575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1771.730548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1771.759043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1772.291533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_aborted -r 2 [ 1772.332038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_aborted -r 2 [ 1772.378376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_aborted -r 2 [ 1772.413204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_aborted -r 2 [ 1772.457515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_aborted -r 2 [ 1772.521362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_aborted -r 2 [ 1772.572026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_aborted -r 2 [ 1772.633096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_aborted -r 2 [ 1772.687544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1772.717321] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1772.996353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1773.063661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1773.088170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1773.114664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1773.157961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1773.197778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1773.228330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1773.263306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1773.281880] LNet: 87480:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.312940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1773.365383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1773.420485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1773.456396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1773.515559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1773.592074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1774.396734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_no_route -r 2 [ 1774.513667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_no_route -r 2 [ 1774.578994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_no_route -r 2 [ 1774.636385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_no_route -r 2 [ 1774.689860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_no_route -r 2 [ 1774.758722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_no_route -r 2 [ 1774.815408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_no_route -r 2 [ 1774.869360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_no_route -r 2 [ 1774.947673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1774.991926] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1775.387959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1775.429522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1775.480075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1775.528415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1775.580412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1775.628069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1775.674541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1775.725831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1775.747221] LNet: 87601:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.770754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1775.823521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1775.847264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1775.869339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1775.911202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1775.960132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1776.507521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e local_timeout -r 2 [ 1776.550554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e local_timeout -r 2 [ 1776.590636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e local_timeout -r 2 [ 1776.636305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e local_timeout -r 2 [ 1776.702733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e local_timeout -r 2 [ 1776.812684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e local_timeout -r 2 [ 1776.904527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e local_timeout -r 2 [ 1776.962511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e local_timeout -r 2 [ 1777.022800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1777.362851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1777.403674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1777.466365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1777.531568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1777.649701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1777.689355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1777.728425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1777.775119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1777.815090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1777.894933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1778.002726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1778.066491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1778.124180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1778.242044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1779.413304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp -m GET -e remote_dropped -r 2 [ 1779.482733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.138@tcp1 -m GET -e remote_dropped -r 2 [ 1779.576805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp -m GET -e remote_dropped -r 2 [ 1779.652585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp -d 192.168.206.38@tcp1 -m GET -e remote_dropped -r 2 [ 1779.698387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp -m GET -e remote_dropped -r 2 [ 1779.758606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.38@tcp1 -d 192.168.206.138@tcp1 -m GET -e remote_dropped -r 2 [ 1779.808549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp -m GET -e remote_dropped -r 2 [ 1779.895395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.38@tcp1 -d 192.168.206.38@tcp1 -m GET -e remote_dropped -r 2 [ 1779.988395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.38@tcp 192.168.206.138@tcp [ 1780.041792] LNet: There was an unexpected network error while writing to 192.168.206.138: rc = -22 [ 1780.048604] LNet: Skipped 1 previous similar message [ 1780.051514] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.138@tcp [ 1780.353442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1780.408650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1780.476676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1780.528801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1780.587421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1780.657541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1780.721378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1780.796416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1780.837133] LNet: 87841:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.847598] LNet: 87841:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1780.896632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1780.962560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1781.061597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1781.138397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1781.192964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1781.294459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1786.731331] LNet: 87916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.753945] LNetError: 87916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.766275] LNet: Removed LNI 192.168.206.38@tcp [ 1786.793929] LNet: Removed LNI 192.168.206.38@tcp1 [ 1787.293979] Key type .llcrypt unregistered [ 1787.299283] Key type ._llcrypt unregistered [ 1804.308484] Key type ._llcrypt registered [ 1804.314168] Key type .llcrypt registered [ 1805.592872] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:12:37 (1775218357) [ 1806.067452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.093334] alg: No test for adler32 (adler32-zlib) [ 1806.994964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.035983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.053520] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1808.000856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1808.064254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.38@tcp [ 1808.086483] LNetError: 88895:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.38@tcp to recovery [ 1808.192294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.38@tcp [ 1808.336825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1816.093993] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:12:47 (1775218367) [ 1816.244995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.284892] LNet: 89479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.305909] LNetError: 89479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.344201] LNet: Removed LNI 192.168.206.38@tcp [ 1816.382392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.482214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.520635] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1817.437400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1817.458373] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 1818.588350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.38@tcp [ 1818.610561] LNetError: 89751:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.38@tcp to recovery [ 1819.551619] LNet: 1 local NIs in recovery (showing 1): 192.168.206.38@tcp [ 1819.553874] LNet: There was an unexpected network error while writing to 192.168.206.38: rc = -22 [ 1819.894961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.38@tcp [ 1829.144769] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:13:00 (1775218380) [ 1831.058379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg638-server [ 1832.983653] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:13:04 (1775218384) [ 1835.152642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg638-server [ 1837.164820] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:13:08 (1775218388) [ 1838.903647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg638-server [ 1840.594694] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:13:12 (1775218392) [ 1840.780844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.831148] LNet: 91382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.844235] LNetError: 91382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.852793] LNet: Removed LNI 192.168.206.38@tcp [ 1840.884679] LNet: Removed LNI 192.168.206.38@tcp1 [ 1840.926818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.993600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1841.029048] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1841.943397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1841.992485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1842.041381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1842.115602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1842.155617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1842.211486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1849.111763] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:13:20 (1775218400) [ 1849.221082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.241546] LNet: 92110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.250944] LNetError: 92110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.258416] LNetError: 92110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1849.278093] LNet: Removed LNI 192.168.206.38@tcp [ 1849.290308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.329292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1849.362067] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1856.894407] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:13:28 (1775218408) [ 1857.292081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.317054] LNet: 92815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.323302] LNetError: 92815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.341792] LNet: Removed LNI 192.168.206.38@tcp [ 1857.369085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.412886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1857.430392] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1867.145257] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:13:38 (1775218418) [ 1868.807694] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1871.187378] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:13:42 (1775218422) [ 1871.750977] LNet: 93850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.756277] LNetError: 93850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.774624] LNet: Removed LNI 192.168.206.38@tcp [ 1872.474938] Key type .llcrypt unregistered [ 1872.479346] Key type ._llcrypt unregistered [ 1879.863178] Key type ._llcrypt registered [ 1879.866596] Key type .llcrypt registered [ 1880.063703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.073671] alg: No test for adler32 (adler32-zlib) [ 1881.075854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1881.112600] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1890.542817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 1896.024122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.138@tcp [ 1896.439604] LNet: 94269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.444076] LNetError: 94269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.470381] LNet: Removed LNI 192.168.206.38@tcp [ 1896.857176] Key type .llcrypt unregistered [ 1896.858840] Key type ._llcrypt unregistered [ 1908.703602] Key type ._llcrypt registered [ 1908.706676] Key type .llcrypt registered [ 1910.036534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1912.062963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1913.616810] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:14:25 (1775218465) [ 1914.209142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.230908] alg: No test for adler32 (adler32-zlib) [ 1915.209625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.309365] LNet: 95356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.047413] Key type .llcrypt unregistered [ 1916.050825] Key type ._llcrypt unregistered [ 1922.739821] Key type ._llcrypt registered [ 1922.742786] Key type .llcrypt registered [ 1923.046326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1923.081490] alg: No test for adler32 (adler32-zlib) [ 1924.189304] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1924.195350] LNet: 95628:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1924.208353] LNet: 95628:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1924.266769] LNetError: 95640:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.40@tcp is being used as a gateway but routing feature is not turned on [ 1924.292280] LNetError: 95640:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.40@tcp has gone from up to down [ 1924.797026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.828893] LNet: 95686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.838633] LNetError: 95686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.864437] LNet: Removed LNI 192.168.206.38@tcp [ 1924.905175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.009034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1925.052288] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1925.061578] LNet: 95693:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1925.073344] LNet: 95693:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1925.088749] LNetError: 95693:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.206.40@tcp is a gateway. Can not delete it [ 1925.100642] LNetError: 95693:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.206.40@tcp is a router. Can not be deleted [ 1925.124959] LNet: 95693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1925.142935] LNetError: 95690:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.42@tcp is being used as a gateway but routing feature is not turned on [ 1925.165941] LNetError: 95690:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1925.189661] LNetError: 95690:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.42@tcp has gone from up to down [ 1925.203332] LNetError: 95690:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1933.114863] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:14:44 (1775218484) [ 1935.451738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg638-server [ 1936.762847] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:14:48 (1775218488) [ 1938.273756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg638-server [ 1940.232373] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:14:51 (1775218491) [ 1942.085160] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1944.088525] Key type lgssc registered [ 1945.636521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.203630] Key type lgssc unregistered [ 1969.588170] LNet: 98115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.597498] LNetError: 98115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.624256] LNet: Removed LNI 192.168.206.38@tcp [ 1970.213666] Key type .llcrypt unregistered [ 1970.215902] Key type ._llcrypt unregistered [ 1984.668779] Key type ._llcrypt registered [ 1984.672785] Key type .llcrypt registered [ 1986.085984] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:15:37 (1775218537) [ 1986.519728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.550951] alg: No test for adler32 (adler32-zlib) [ 1987.500251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.578065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1987.640776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1987.696620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1987.776306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1993.748532] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:15:45 (1775218545) [ 1993.878263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.906405] LNet: 99514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.938646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.047902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.088037] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1995.283756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.307775] LNet: 99665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.321734] LNetError: 99665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.359328] LNet: Removed LNI 192.168.206.38@tcp [ 1995.420656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.529920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1995.592535] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 1996.221909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.242935] LNetError: 99694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.267949] LNet: Removed LNI 192.168.206.38@tcp [ 1996.290193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.345856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1996.372604] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2002.641487] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:15:54 (1775218554) [ 2002.806697] LNet: 100283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.812860] LNet: 100283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2002.833446] LNetError: 100283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.858929] LNet: Removed LNI 192.168.206.38@tcp [ 2003.406749] Key type .llcrypt unregistered [ 2003.412574] Key type ._llcrypt unregistered [ 2009.468395] Key type ._llcrypt registered [ 2009.472048] Key type .llcrypt registered [ 2009.635532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2009.644933] alg: No test for adler32 (adler32-zlib) [ 2010.538046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.605129] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2010.608236] LNet: *** cfs_fail_loc=e003, val=100*** [ 2011.348127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 2011.368103] LNet: *** cfs_fail_loc=e003, val=100*** [ 2011.370914] LNet: Skipped 2 previous similar messages [ 2011.382206] Ping reply received with mapping: [ 2011.389956] found metadata reply NI[192.168.206.38@tcp] [ 2011.395199] 26 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2011.401396] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2011.405278] 5a 5a 5a 5a 5a 5a 5a 5a [ 2011.543742] LNet: 100681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.553458] LNetError: 100681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.575249] LNet: Removed LNI 192.168.206.38@tcp [ 2012.316414] Key type .llcrypt unregistered [ 2012.321429] Key type ._llcrypt unregistered [ 2024.518897] Key type ._llcrypt registered [ 2024.524279] Key type .llcrypt registered [ 2025.894793] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:16:17 (1775218577) [ 2026.443298] Key type .llcrypt unregistered [ 2026.446244] Key type ._llcrypt unregistered [ 2033.038699] Key type ._llcrypt registered [ 2033.040835] Key type .llcrypt registered [ 2033.294258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2033.316912] alg: No test for adler32 (adler32-zlib) [ 2034.467659] LNetError: 101727:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2034.865513] Key type .llcrypt unregistered [ 2034.871723] Key type ._llcrypt unregistered [ 2040.491694] Key type ._llcrypt registered [ 2040.498368] Key type .llcrypt registered [ 2041.974793] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:16:33 (1775218593) [ 2042.571836] Key type .llcrypt unregistered [ 2042.574627] Key type ._llcrypt unregistered [ 2049.951860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2049.959829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2051.511985] Key type ._llcrypt registered [ 2051.514737] Key type .llcrypt registered [ 2051.831158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.859950] alg: No test for adler32 (adler32-zlib) [ 2052.820325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.893648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2063.366891] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:16:54 (1775218614) [ 2063.536381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.575800] LNet: 103494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.586173] LNetError: 103494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2063.618896] LNet: Removed LNI 10.1.2.3@tcp [ 2063.651597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.772505] Key type .llcrypt unregistered [ 2064.775863] Key type ._llcrypt unregistered [ 2071.922169] Key type ._llcrypt registered [ 2071.930373] Key type .llcrypt registered [ 2072.241066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2072.259888] alg: No test for adler32 (adler32-zlib) [ 2073.249474] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2073.344681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 2073.514988] LNet: 103786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.528376] LNetError: 103786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.542953] LNet: Removed LNI 192.168.206.38@tcp [ 2074.232955] Key type .llcrypt unregistered [ 2074.234796] Key type ._llcrypt unregistered [ 2086.858410] Key type ._llcrypt registered [ 2086.866122] Key type .llcrypt registered [ 2088.512179] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:17:19 (1775218639) [ 2089.359441] Key type .llcrypt unregistered [ 2089.365066] Key type ._llcrypt unregistered [ 2095.854622] Key type ._llcrypt registered [ 2095.856121] Key type .llcrypt registered [ 2096.056228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.075906] alg: No test for adler32 (adler32-zlib) [ 2103.253697] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:17:34 (1775218654) [ 2103.363948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.406542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.466959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.499812] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2104.349967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2104.385956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2104.416297] LNet: Removed LNI 192.168.206.38@tcp [ 2104.719665] LNet: 105530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.153674] Key type .llcrypt unregistered [ 2105.160516] Key type ._llcrypt unregistered [ 2118.412701] Key type ._llcrypt registered [ 2118.418500] Key type .llcrypt registered [ 2120.526853] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:17:51 (1775218671) [ 2121.189473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.234875] alg: No test for adler32 (adler32-zlib) [ 2122.198863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.289471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.335766] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2123.152439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 2123.205376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.38@tcp [ 2123.234738] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2123.247950] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2123.270573] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2123, 0, 0 [ 2123.274203] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2123.286383] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2123.300988] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2123.308713] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2123, 0, 0 [ 2123.316628] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2123.321492] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2123.325970] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2123.337401] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2123, 0, 2173 [ 2123.351661] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2123.355879] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2123.361397] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2123.365795] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2123, 0, 2173 [ 2123.370561] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2131.161709] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:18:02 (1775218682) [ 2131.826559] LNet: 107070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.833988] LNetError: 107070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.853455] LNet: Removed LNI 192.168.206.38@tcp [ 2132.651535] Key type .llcrypt unregistered [ 2132.654156] Key type ._llcrypt unregistered [ 2139.883932] Key type ._llcrypt registered [ 2139.886231] Key type .llcrypt registered [ 2140.053384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2140.066694] alg: No test for adler32 (adler32-zlib) [ 2141.064217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2141.105297] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2149.159917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 2157.787974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.825437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2157.871960] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 2159.039710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2159.104551] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2159.112637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2159.125499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2161.530806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2161.566688] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2171.379547] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:18:43 (1775218723) [ 2171.542988] LNet: 108571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.551359] LNetError: 108571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.560653] LNet: Removed LNI 192.168.206.38@tcp [ 2171.570692] LNet: Removed LNI 192.168.206.38@tcp1 [ 2171.584156] LNet: Removed LNI 10.1.2.3@tcp99 [ 2172.288690] Key type .llcrypt unregistered [ 2172.290363] Key type ._llcrypt unregistered [ 2178.087489] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2178.097961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2180.782277] Key type ._llcrypt registered [ 2180.795055] Key type .llcrypt registered [ 2181.062882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2181.097300] alg: No test for adler32 (adler32-zlib) [ 2182.063977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.134125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.158600] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2182.826880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2182.857330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.652654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2184.688515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2184.740952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2184.769252] LNetError: 109321:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2184.816769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2184.866832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2185.270273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2185.369987] LNet: 109341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.380043] LNetError: 109341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.385206] LNet: Removed LNI 192.168.206.38@tcp [ 2185.394338] LNet: Removed LNI 10.1.2.3@tcp [ 2186.090897] Key type .llcrypt unregistered [ 2186.093070] Key type ._llcrypt unregistered [ 2197.804286] Key type ._llcrypt registered [ 2197.808338] Key type .llcrypt registered [ 2199.269156] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:19:10 (1775218750) [ 2199.940443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.954579] alg: No test for adler32 (adler32-zlib) [ 2200.954685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.009252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2201.038669] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2201.804287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.38@tcp [ 2201.920672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg638-client@tcp [ 2208.421752] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:19:20 (1775218760) [ 2208.679551] LNet: 110874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.684951] LNetError: 110874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.716773] LNet: Removed LNI 192.168.206.38@tcp [ 2210.383557] Key type .llcrypt unregistered [ 2210.387935] Key type ._llcrypt unregistered [ 2216.902393] Key type ._llcrypt registered [ 2216.908600] Key type .llcrypt registered [ 2217.099887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.115468] alg: No test for adler32 (adler32-zlib) [ 2218.011552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2218.039546] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2218.067487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2225.347539] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:19:36 (1775218776) [ 2227.005309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2228.566727] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:19:40 (1775218780) [ 2230.039365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2231.837697] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:19:43 (1775218783) [ 2232.087897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.118516] LNet: 112294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.134662] LNetError: 112294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.151712] LNet: Removed LNI 192.168.206.38@tcp [ 2232.182305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.257146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.291169] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2232.926346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2232.969862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2240.033990] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:19:51 (1775218791) [ 2240.246889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.294950] LNet: 113003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.314467] LNetError: 113003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.346671] LNet: Removed LNI 192.168.206.38@tcp [ 2240.396547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.448456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2240.470220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2241.175319] Key type .llcrypt unregistered [ 2241.178558] Key type ._llcrypt unregistered [ 2253.544285] Key type ._llcrypt registered [ 2253.546014] Key type .llcrypt registered [ 2254.873363] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:20:06 (1775218806) [ 2255.485022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.514245] alg: No test for adler32 (adler32-zlib) [ 2256.456399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.527883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.562925] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2257.363920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2257.422883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2264.206839] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:20:15 (1775218815) [ 2264.582451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.609954] LNet: 114559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.616898] LNetError: 114559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.628775] LNet: Removed LNI 192.168.206.38@tcp [ 2264.702985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.742971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2264.778483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2271.973402] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:20:23 (1775218823) [ 2272.214178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.246846] LNet: 115140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.288032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.370221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2272.593857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2272.630936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2272.697659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2273.062600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2273.096643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2273.179702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.210993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.296636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2273.707027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2273.769115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2273.858757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2274.413837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2274.444954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2282.019496] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:20:33 (1775218833) [ 2282.190873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.223726] LNet: 115771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.231457] LNet: 115771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2282.281232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.325218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.346361] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2282.984259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2283.016409] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2284.214910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2284.303626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2284.678496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.714221] LNet: 116053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.720032] LNetError: 116053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.731280] LNet: Removed LNI 192.168.206.38@tcp [ 2284.751154] LNet: Removed LNI 192.168.206.38@tcp2 [ 2284.783979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.899993] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2284.910671] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2284.918691] LNet: 116059:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.435966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2285.802904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.831813] LNetError: 116097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.845279] LNetError: 116097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2285.873569] LNet: Removed LNI 192.168.206.38@tcp [ 2285.885157] LNet: Removed LNI 192.168.206.38@tcp2 [ 2285.918591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.993592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.026451] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2287.183266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2287.233848] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2288.165694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2288.254223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2288.353422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2288.630765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.678257] LNetError: 116381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.690302] LNetError: 116381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2288.704400] LNet: Removed LNI 192.168.206.38@tcp [ 2289.769860] LNet: Removed LNI 192.168.206.38@tcp2 [ 2289.823548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.903933] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2289.909802] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2289.916968] LNet: 116387:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2290.373713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2290.590495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.613821] LNet: 116424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.616845] LNet: 116424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2290.620721] LNet: Removed LNI 192.168.206.38@tcp [ 2291.687101] LNet: Removed LNI 192.168.206.38@tcp2 [ 2291.734855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.819527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2291.865726] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2292.657338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2292.686809] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2293.838754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2293.912588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2294.272835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.339587] LNetError: 116704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.349698] LNetError: 116704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2294.367977] LNet: Removed LNI 192.168.206.38@tcp [ 2294.391420] LNet: Removed LNI 192.168.206.38@tcp2 [ 2294.409773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.522501] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2294.534834] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2294.550836] LNet: 116710:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.106164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2295.394758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.479356] LNet: Removed LNI 192.168.206.38@tcp [ 2296.491781] LNet: Removed LNI 192.168.206.38@tcp2 [ 2296.549614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.640992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.682919] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2297.834179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2297.864342] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2298.901800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2299.002449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2299.082259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2299.333920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2299.365401] LNet: 117032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.372476] LNet: 117032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2300.448368] LNet: Removed LNI 192.168.206.38@tcp [ 2301.481349] LNet: Removed LNI 192.168.206.38@tcp2 [ 2301.550450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.659267] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2301.670546] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2301.679472] LNet: 117038:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.356434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2302.575811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.612669] LNetError: 117075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.627757] LNetError: 117075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2302.641215] LNet: Removed LNI 192.168.206.38@tcp [ 2303.656300] LNet: Removed LNI 192.168.206.38@tcp2 [ 2303.690452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.753651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.798560] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2304.594156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2304.616399] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2305.821929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2305.846463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2305.889050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2306.234985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.295406] LNet: Removed LNI 192.168.206.38@tcp [ 2307.314487] LNet: Removed LNI 192.168.206.38@tcp2 [ 2307.408165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.487958] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2307.504152] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2307.536552] LNet: 117378:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2308.228271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2308.913586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.983679] LNet: Removed LNI 192.168.206.38@tcp [ 2310.000921] LNet: Removed LNI 192.168.206.38@tcp2 [ 2310.064815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.188731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.234061] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2311.337900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2311.359041] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2312.848554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2312.902978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2312.951282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2313.042250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2313.491303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.529430] LNet: Removed LNI 192.168.206.38@tcp [ 2315.562132] LNet: Removed LNI 192.168.206.38@tcp2 [ 2315.685414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.846611] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2315.864860] LNet: Added LNI 192.168.206.38@tcp2 [8/256/0/180] [ 2315.873526] LNet: 117738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2316.832262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2326.616212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2328.504932] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:21:20 (1775218880) [ 2328.706498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.779087] LNet: 118491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.790961] LNet: 118491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2328.837539] LNetError: 118491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.859636] LNetError: 118491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2328.877272] LNet: Removed LNI 192.168.206.38@tcp [ 2329.959648] LNet: Removed LNI 192.168.206.38@tcp2 [ 2330.024571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.485776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.563829] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2331.747132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2331.807644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2333.064504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.112604] LNet: Removed LNI 192.168.206.38@tcp [ 2333.131525] LNet: Removed LNI 10.1.2.3@tcp [ 2333.168814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.268494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2333.303057] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2333.312653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2334.138468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2334.168344] LNet: Removed LNI 192.168.206.38@tcp [ 2334.188925] LNet: Removed LNI 10.1.2.3@tcp [ 2340.932862] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:21:32 (1775218892) [ 2341.402659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.506388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.697432] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2341.855588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2341.862291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2341.867262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2344.179158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.217893] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2345.486753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2345.533177] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2348.150516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.38@tcp [ 2348.211620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2348.287288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.38@tcp -m PUT -e local_timeout -r 1 [ 2348.375458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2348.480609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2348.498093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2348.810903] LNetError: 119439:0:(lib-move.c:4969:lnet_parse()) 192.168.206.38@tcp, src 192.168.206.38@tcp: Dropping PUT (error -108 looking up sender) [ 2350.879573] LNet: Removed LNI 192.168.206.38@tcp [ 2350.901553] LNet: Removed LNI 10.1.2.3@tcp [ 2351.400136] Key type .llcrypt unregistered [ 2351.406988] Key type ._llcrypt unregistered [ 2365.078489] Key type ._llcrypt registered [ 2365.082533] Key type .llcrypt registered [ 2366.788644] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:21:58 (1775218918) [ 2367.455241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.492228] alg: No test for adler32 (adler32-zlib) [ 2368.378365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.859925] LNet: 120751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.372362] Key type .llcrypt unregistered [ 2369.382134] Key type ._llcrypt unregistered [ 2377.443924] Key type ._llcrypt registered [ 2377.447232] Key type .llcrypt registered [ 2377.795691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2377.851784] alg: No test for adler32 (adler32-zlib) [ 2378.833683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2378.873242] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2388.963627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 2389.168555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2390.191865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.138@tcp [ 2390.511947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2390.578195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2390.639978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2390.681437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2390.725322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2390.785594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2390.837275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2390.890763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2390.926100] LNet: 121201:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2390.984277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2391.054044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2391.135784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2391.201462] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.138@tcp [ 2391.235336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2391.356416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2391.467308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2392.451316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2392.757898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2392.804541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2392.884518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2392.939270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2393.041181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2393.097787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2393.158671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2393.209724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2393.239255] LNet: 121319:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.286242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2393.373977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2393.420416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2393.516587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2393.568530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2393.627617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2394.885871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.138@tcp [ 2395.152858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2395.183788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2395.230447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2395.287719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2395.334902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2395.397269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2395.429720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2395.459995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2395.480174] LNet: 121436:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2395.499116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2395.524521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2395.567628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2395.627292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2395.776898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2395.851711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2397.043441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2397.507695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2397.585390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2397.662438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2397.729772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2397.799496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2397.872722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2397.944913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2398.003672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2398.036647] LNet: 121553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2398.066133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2398.102988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2398.137806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2398.178730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2398.212117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2398.247595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2398.623305] LNet: 121599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.628468] LNetError: 121599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.663467] LNet: Removed LNI 192.168.206.38@tcp [ 2400.031758] Key type .llcrypt unregistered [ 2400.041342] Key type ._llcrypt unregistered [ 2423.640928] Key type ._llcrypt registered [ 2423.646213] Key type .llcrypt registered [ 2425.029569] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:22:56 (1775218976) [ 2425.500687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.546640] alg: No test for adler32 (adler32-zlib) [ 2426.475814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.941462] LNet: 122488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.642438] Key type .llcrypt unregistered [ 2427.646987] Key type ._llcrypt unregistered [ 2434.626729] Key type ._llcrypt registered [ 2434.632443] Key type .llcrypt registered [ 2434.861986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2434.878233] alg: No test for adler32 (adler32-zlib) [ 2435.837294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2435.873562] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2443.833741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.138@tcp [ 2453.933369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.990578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2454.022610] LNet: Added LNI 192.168.206.38@tcp1 [8/256/0/180] [ 2455.197721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2455.856308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.138@tcp [ 2456.079807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2456.110713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2456.161645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2456.185722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.210247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2456.254759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2456.287867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2456.331730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2456.351401] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.138@tcp [ 2456.362280] LNet: 123140:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2456.418630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2456.483673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2456.537326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2456.582848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2456.617325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2456.676818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2457.368192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2457.637624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2457.677954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2457.707971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2457.752820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2457.789119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2457.817091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2457.847661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2457.893568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2457.924357] LNet: 123257:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2457.965814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2458.007596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2458.059421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2458.093748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2458.129844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2458.163463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2458.792507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.138@tcp [ 2459.071480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2459.131116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2459.161362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2459.185631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2459.209449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2459.233492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2459.260963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2459.291593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2459.324802] LNet: 123374:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2459.360991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2459.399295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2459.454911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2459.505196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2459.550458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2459.597673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2460.342316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2460.560249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.583700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2460.618424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.678489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.723452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.772747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2460.810452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2460.841406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2460.873645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2460.907692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2460.947841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2460.982539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2461.010883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2461.041562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2466.566991] LNet: 123572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.581958] LNetError: 123572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.589850] LNet: Removed LNI 192.168.206.38@tcp [ 2466.633708] LNet: Removed LNI 192.168.206.38@tcp1 [ 2467.197337] Key type .llcrypt unregistered [ 2467.201695] Key type ._llcrypt unregistered [ 2486.638171] Key type ._llcrypt registered [ 2486.640180] Key type .llcrypt registered [ 2488.367633] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:23:59 (1775219039) [ 2496.937405] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:24:08 (1775219048) [ 2505.681367] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:24:16 (1775219056) [ 2514.576759] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:24:26 (1775219066) [ 2523.801251] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:24:35 (1775219075) [ 2532.272302] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:24:44 (1775219084) [ 2541.612571] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:24:53 (1775219093) [ 2551.960820] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:25:03 (1775219103) [ 2561.539933] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:25:13 (1775219113) [ 2570.859381] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:25:21 (1775219121) [ 2578.456129] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:25:29 (1775219129) [ 2589.165783] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:25:40 (1775219140) [ 2597.341491] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:25:48 (1775219148) [ 2605.277619] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:25:57 (1775219157) [ 2612.772254] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:26:04 (1775219164) [ 2620.435618] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:26:11 (1775219171) [ 2627.730972] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:26:19 (1775219179) [ 2637.038298] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:26:28 (1775219188) [ 2637.514770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2637.538578] alg: No test for adler32 (adler32-zlib) [ 2638.587053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2638.864801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2638.895494] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2640.687767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2640.726315] LNet: 134287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.741574] LNetError: 134287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2640.763255] LNet: Removed LNI 192.168.206.38@tcp [ 2640.787913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2640.883407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2640.948118] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2648.708067] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:26:40 (1775219200) [ 2657.054102] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:26:48 (1775219208) [ 2665.055641] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:26:56 (1775219216) [ 2672.699676] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:27:04 (1775219224) [ 2680.524622] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:27:12 (1775219232) [ 2689.040399] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:27:20 (1775219240) [ 2699.394872] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:27:30 (1775219250) [ 2709.859921] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:27:41 (1775219261) [ 2718.467774] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:27:50 (1775219270) [ 2727.386387] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:27:58 (1775219278) [ 2735.326176] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:28:06 (1775219286) [ 2744.776063] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:28:15 (1775219295) [ 2753.434145] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2539 sec ========== 08:28:24 (1775219304) [ 2754.236827] LNet: 141757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2754.241979] LNetError: 141757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2754.254398] LNet: Removed LNI 192.168.206.38@tcp [ 2755.445475] Key type .llcrypt unregistered [ 2755.446753] Key type ._llcrypt unregistered [ 2771.177241] Key type ._llcrypt registered [ 2771.180877] Key type .llcrypt registered [ 2771.604665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2771.626837] alg: No test for adler32 (adler32-zlib) [ 2772.786819] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 2772.981964] LNet: Added LNI 192.168.206.38@tcp [8/256/0/180] [ 2774.695642] Key type lgssc registered [ 2775.948785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2840.747553] Lustre: Mounted lustre-client [ 2846.607631] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2861.192488] LustreError: 144215:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9e99113a0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2861.256731] Lustre: Unmounted lustre-client [ 2885.977854] Key type lgssc unregistered [ 2886.475738] LNet: 144696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2886.495967] LNetError: 144696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.531037] LNet: Removed LNI 192.168.206.38@tcp [ 2887.628161] Key type .llcrypt unregistered [ 2887.629503] Key type ._llcrypt unregistered