[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471774086 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007418] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013092] *** VALIDATE selinux *** [ 0.020290] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024278] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027135] *** VALIDATE tmpfs *** [ 0.028374] *** VALIDATE proc *** [ 0.029250] *** VALIDATE cgroup *** [ 0.031011] *** VALIDATE cgroup2 *** [ 0.032253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035025] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039407] debug: unmapping init [mem 0xffffffff89659000-0xffffffff89660fff] [ 0.042205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043769] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.044934] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.051209] rcu: Hierarchical SRCU implementation. [ 0.053414] smp: Bringing up secondary CPUs ... [ 0.054505] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.062475] smp: Brought up 1 node, 4 CPUs [ 0.063905] smpboot: Max logical packages: 1 [ 0.064034] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.207916] node 0 deferred pages initialised in 141ms [ 0.211166] devtmpfs: initialized [ 0.212211] x86/mm: Memory block size: 128MB [ 0.216241] gcov: version magic: 0x41383552 [ 0.218264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.222093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.224304] pinctrl core: initialized pinctrl subsystem [ 0.225176] [ 0.225559] ************************************************************* [ 0.228013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.230010] ** ** [ 0.232015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234015] ** ** [ 0.237015] ** This means that this kernel is built to expose internal ** [ 0.239018] ** IOMMU data structures, which may compromise security on ** [ 0.241011] ** your system. ** [ 0.242008] ** ** [ 0.244016] ** If you see this message and you are not debugging the ** [ 0.246016] ** kernel, report this immediately to your vendor! ** [ 0.248013] ** ** [ 0.250010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252011] ************************************************************* [ 0.253663] NET: Registered protocol family 16 [ 0.255341] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257159] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.259137] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262121] cpuidle: using governor menu [ 0.263770] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.266557] PCI: Using configuration type 1 for base access [ 0.268129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.276046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.277037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280238] cryptd: max_cpu_qlen set to 1000 [ 0.283224] ACPI: Added _OSI(Module Device) [ 0.284012] ACPI: Added _OSI(Processor Device) [ 0.285015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.286016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289102] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.295569] ACPI: Interpreter enabled [ 0.298069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300024] ACPI: Using IOAPIC for interrupt routing [ 0.302108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.304304] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312158] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.316018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322391] acpiphp: Slot [2] registered [ 0.323101] acpiphp: Slot [5] registered [ 0.324125] acpiphp: Slot [6] registered [ 0.326131] acpiphp: Slot [3] registered [ 0.327118] acpiphp: Slot [4] registered [ 0.328174] acpiphp: Slot [7] registered [ 0.330062] acpiphp: Slot [8] registered [ 0.331072] acpiphp: Slot [9] registered [ 0.332065] acpiphp: Slot [10] registered [ 0.333000] acpiphp: Slot [11] registered [ 0.333000] acpiphp: Slot [12] registered [ 0.333121] acpiphp: Slot [13] registered [ 0.334000] acpiphp: Slot [14] registered [ 0.335060] acpiphp: Slot [15] registered [ 0.336010] acpiphp: Slot [16] registered [ 0.336967] acpiphp: Slot [17] registered [ 0.338091] acpiphp: Slot [18] registered [ 0.340094] acpiphp: Slot [19] registered [ 0.340992] acpiphp: Slot [20] registered [ 0.341057] acpiphp: Slot [21] registered [ 0.342047] acpiphp: Slot [22] registered [ 0.343054] acpiphp: Slot [23] registered [ 0.344024] acpiphp: Slot [24] registered [ 0.344965] acpiphp: Slot [25] registered [ 0.346096] acpiphp: Slot [26] registered [ 0.346925] acpiphp: Slot [27] registered [ 0.349077] acpiphp: Slot [28] registered [ 0.350008] acpiphp: Slot [29] registered [ 0.350832] acpiphp: Slot [30] registered [ 0.351073] acpiphp: Slot [31] registered [ 0.353057] PCI host bridge to bus 0000:00 [ 0.354011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.357017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.359023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.361026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.364201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.370075] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377512] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.380056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.384015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.386016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397181] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.402013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.410016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417584] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471761] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.483024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.504021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.516884] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520542] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.524545] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.527488] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.530278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.537071] iommu: Default domain type: Passthrough [ 0.541000] SCSI subsystem initialized [ 0.543179] ACPI: bus type USB registered [ 0.545173] usbcore: registered new interface driver usbfs [ 0.548104] usbcore: registered new interface driver hub [ 0.550071] usbcore: registered new device driver usb [ 0.552216] pps_core: LinuxPPS API ver. 1 registered [ 0.554012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.556063] PTP clock support registered [ 0.559034] EDAC MC: Ver: 3.0.0 [ 0.561025] PCI: Using ACPI for IRQ routing [ 0.563017] NetLabel: Initializing [ 0.564011] NetLabel: domain hash size = 128 [ 0.565011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.566120] NetLabel: unlabeled traffic allowed by default [ 0.569281] vgaarb: loaded [ 0.571341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.573020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.582126] clocksource: Switched to clocksource kvm-clock [ 0.678502] VFS: Disk quotas dquot_6.6.0 [ 0.680114] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.681761] *** VALIDATE ramfs *** [ 0.682606] *** VALIDATE hugetlbfs *** [ 0.683678] pnp: PnP ACPI init [ 0.686166] pnp: PnP ACPI: found 6 devices [ 0.703552] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.705772] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.706962] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.708221] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.709495] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710977] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.713509] NET: Registered protocol family 2 [ 0.715523] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.719905] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.723399] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.728386] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.731500] TCP: Hash tables configured (established 65536 bind 65536) [ 0.734510] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.737573] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.739634] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.742146] NET: Registered protocol family 1 [ 0.744963] RPC: Registered named UNIX socket transport module. [ 0.747273] RPC: Registered udp transport module. [ 0.748897] RPC: Registered tcp transport module. [ 0.750576] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.752801] NET: Registered protocol family 44 [ 0.754347] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.756510] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.758362] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.760331] PCI: CLS 0 bytes, default 64 [ 0.761971] Unpacking initramfs... [ 2.434599] debug: unmapping init [mem 0xffff8e3a3cc64000-0xffff8e3a3ffcffff] [ 2.439126] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.441546] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444919] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.977852] Initialise system trusted keyrings [ 2.980373] Key type blacklist registered [ 2.982071] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990195] zbud: loaded [ 2.993183] *** VALIDATE nfs *** [ 2.994420] *** VALIDATE nfs4 *** [ 2.996212] pstore: using deflate compression [ 3.000400] Platform Keyring initialized [ 3.145430] NET: Registered protocol family 38 [ 3.148380] Key type asymmetric registered [ 3.151700] Asymmetric key parser 'x509' registered [ 3.154803] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.158355] io scheduler mq-deadline registered [ 3.159886] io scheduler kyber registered [ 3.161593] io scheduler bfq registered [ 3.163643] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.167255] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.170325] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.173281] ACPI: Power Button [PWRF] [ 3.180222] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.188782] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.204971] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.236240] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.266836] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.271940] Non-volatile memory driver v1.3 [ 3.273579] Linux agpgart interface v0.103 [ 3.306772] virtio_blk virtio1: [vda] 134008 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.309226] vda: detected capacity change from 0 to 68612096 [ 3.323533] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.326413] vdb: detected capacity change from 0 to 1073741824 [ 3.347569] libphy: Fixed MDIO Bus: probed [ 3.354797] usbcore: registered new interface driver usbserial_generic [ 3.357816] usbserial: USB Serial support registered for generic [ 3.360298] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.366205] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.368254] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.370646] mousedev: PS/2 mouse device common for all mice [ 3.374781] rtc_cmos 00:05: RTC can wake from S4 [ 3.378168] rtc_cmos 00:05: registered as rtc0 [ 3.381305] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.384341] intel_pstate: CPU model not supported [ 3.387724] hid: raw HID events driver (C) Jiri Kosina [ 3.389812] usbcore: registered new interface driver usbhid [ 3.392105] usbhid: USB HID core driver [ 3.393854] drop_monitor: Initializing network drop monitor service [ 3.396531] Initializing XFRM netlink socket [ 3.397599] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.403220] NET: Registered protocol family 10 [ 3.407549] Segment Routing with IPv6 [ 3.413573] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.416032] NET: Registered protocol family 17 [ 3.420274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.425576] mpls_gso: MPLS GSO support [ 3.432425] RAS: Correctable Errors collector initialized. [ 3.434490] AVX version of gcm_enc/dec engaged. [ 3.437807] AES CTR mode by8 optimization enabled [ 3.573834] sched_clock: Marking stable (3573673966, 0)->(4358090312, -784416346) [ 3.578814] registered taskstats version 1 [ 3.581556] Loading compiled-in X.509 certificates [ 3.584290] zswap: loaded using pool lzo/zbud [ 3.627890] Key type big_key registered [ 3.645882] Key type encrypted registered [ 3.648416] ima: No TPM chip found, activating TPM-bypass! [ 3.653295] ima: Allocated hash algorithm: sha1 [ 3.656479] ima: No architecture policies found [ 3.660144] evm: Initialising EVM extended attributes: [ 3.663315] evm: security.selinux [ 3.664541] evm: security.ima [ 3.666453] evm: security.capability [ 3.668182] evm: HMAC attrs: 0x1 [ 3.671406] rtc_cmos 00:05: setting system clock to 2026-04-24 15:09:08 UTC (1777043348) [ 3.680957] debug: unmapping init [mem 0xffffffff8a603000-0xffffffff8a7fffff] [ 3.684860] debug: unmapping init [mem 0xffffffff89382000-0xffffffff89658fff] [ 3.697867] Write protecting the kernel read-only data: 28672k [ 3.703688] debug: unmapping init [mem 0xffffffff87a03000-0xffffffff87bfffff] [ 3.708366] debug: unmapping init [mem 0xffffffff88314000-0xffffffff883fffff] [ 3.750670] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.759617] systemd[1]: Detected virtualization kvm. [ 3.764298] systemd[1]: Detected architecture x86-64. [ 3.766552] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.800522] systemd[1]: No hostname configured. [ 3.803974] systemd[1]: Set hostname to . [ 3.809213] random: systemd: uninitialized urandom read (16 bytes read) [ 3.814161] systemd[1]: Initializing machine ID from random generator. [ 4.531443] random: systemd: uninitialized urandom read (16 bytes read) [ 4.533971] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.542262] random: systemd: uninitialized urandom read (16 bytes read) [ 4.545269] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.549295] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.154421] device-mapper: uevent: version 1.0.3 [ 6.156575] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.161051] virtio_net virtio0 ens2: renamed from eth0 [ 9.914029] scsi host0: ata_piix [ 10.327844] scsi host1: ata_piix [ 10.329275] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.336263] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 14.866614] random: fast init done [ 15.189544] random: crng init done [ 15.190673] random: 7 urandom warning(s) missed due to ratelimiting [ 17.421659] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.608949] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.356710] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.500813] SELinux: Disabled at runtime. [ 23.706606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.740973] systemd[1]: Detected virtualization kvm. [ 23.742850] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.631396] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.637249] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.645684] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.655168] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.660453] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.670369] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.688980] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice sy[ 25.910464] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS stem-getty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.145701] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.205128] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.320905] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.852164] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.988664] EDAC sbridge: Ver: 1.1.2 [ 32.547340] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 33.432759] NFS: Registering the id_resolver key type [ 33.438517] Key type id_resolver registered [ 33.441280] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg250-client login: [ 86.576763] libcfs: loading out-of-tree module taints kernel. [ 86.692178] Key type ._llcrypt registered [ 86.696820] Key type .llcrypt registered [ 86.993364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 87.001758] alg: No test for adler32 (adler32-zlib) [ 88.236410] Lustre: Lustre: Build Version: 2.17.52_54_g344a3c0 [ 88.791340] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 90.447159] Key type lgssc registered [ 91.350325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 172.179938] Lustre: Mounted lustre-client [ 175.599795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 191.480664] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing check_logdir /tmp/testlogs/ [ 195.369608] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing yml_node [ 198.076855] Lustre: DEBUG MARKER: Client: 2.17.52.54 [ 198.111158] Lustre: lustre-OST0000-osc-ffff8e3a83f22000: disconnect after 24s idle [ 199.904319] Lustre: DEBUG MARKER: MDS: 2.17.52.54 [ 201.610748] Lustre: DEBUG MARKER: OSS: 2.17.52.54 [ 202.869837] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Apr 24 11:12:26 EDT 2026 [ 212.153418] LustreError: 7909:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8e3a83f22000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 212.179275] Lustre: Unmounted lustre-client [ 228.613496] Key type lgssc unregistered [ 228.836632] LNet: 8353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 228.847136] LNetError: 8353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 228.859656] LNet: Removed LNI 192.168.202.50@tcp [ 229.333463] Key type .llcrypt unregistered [ 229.334919] Key type ._llcrypt unregistered [ 244.713124] Key type ._llcrypt registered [ 244.714666] Key type .llcrypt registered [ 244.858670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 244.867643] alg: No test for adler32 (adler32-zlib) [ 245.704214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 245.737377] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 245.746848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 248.133967] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 248.225126] LNet: 9229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 248.229512] LNetError: 9229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 248.241484] LNet: Removed LNI 192.168.202.50@tcp [ 248.582547] Key type .llcrypt unregistered [ 248.583834] Key type ._llcrypt unregistered [ 252.133688] Key type ._llcrypt registered [ 252.135491] Key type .llcrypt registered [ 252.941927] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:13:16 (1777043596) [ 253.123072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 253.131480] alg: No test for adler32 (adler32-zlib) [ 253.988343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.043978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.059586] LNet: 9643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.074500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.111673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 254.122862] LNet: 9650:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 258.039522] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:13:22 (1777043602) [ 258.135438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.158139] LNet: 10223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.197136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.258620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 262.633226] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:13:26 (1777043606) [ 262.729101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.751501] LNet: 10808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.785421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.842884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 267.019563] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:13:30 (1777043610) [ 267.070360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.088209] LNet: 11389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 267.105622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.160927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 271.530412] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:13:35 (1777043615) [ 271.586569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.604829] LNet: 11973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.624682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.701106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 272.004538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 275.705363] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:13:39 (1777043619) [ 275.882670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.903767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.945524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 276.141603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 276.335864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 279.830193] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:13:43 (1777043623) [ 279.946601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.956637] LNet: 13165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.960369] LNet: 13165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 279.977799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.013400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 283.519362] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:13:47 (1777043627) [ 283.667649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.683821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.720251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 283.739152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 283.927772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 283.953144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 284.144141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 284.177858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 284.385687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 284.404511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 284.626086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 284.641647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 284.889730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 284.912490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 285.172452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 285.193224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 285.408245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 285.425275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 285.626975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 285.653612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 285.857582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 285.872828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 286.116271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 286.129822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 286.393743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 286.421590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 286.645651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 286.667462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 290.467902] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:13:54 (1777043634) [ 290.541459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.565455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.600904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 290.621914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 294.599737] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:13:58 (1777043638) [ 294.726707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.757792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.816561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 294.844460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 298.527630] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:14:02 (1777043642) [ 298.571094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.589145] LNet: 15594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.599518] LNet: 15594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 298.628052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.696468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 298.709916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 302.044968] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:14:06 (1777043646) [ 302.144224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.161852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.201349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 302.231910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 305.612666] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:14:09 (1777043649) [ 305.727859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.759045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.810326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 305.840539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 309.467920] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:14:13 (1777043653) [ 309.609980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.633106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.665866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 309.697458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 313.188375] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:14:17 (1777043657) [ 313.324688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.340025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.377771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 313.590301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 317.215112] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:14:21 (1777043661) [ 317.333300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.357063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.456419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.586208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 321.024219] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:14:25 (1777043665) [ 321.148494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.169109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.237777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 321.350951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 325.462432] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:14:29 (1777043669) [ 325.535848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.570288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.930077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 326.086209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 330.256694] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:14:34 (1777043674) [ 330.311829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.338902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.394782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.434455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 334.328072] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:14:38 (1777043678) [ 334.479056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.494864] LNet: 21017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.499937] LNet: 21017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 334.517669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.554346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 334.599229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 338.903482] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:14:42 (1777043682) [ 338.962175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.993159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.059173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 342.657347] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:14:46 (1777043686) [ 342.771218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.800640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.863600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 346.574687] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:14:50 (1777043690) [ 346.726580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.759391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.816436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 347.062126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 350.597641] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:14:54 (1777043694) [ 350.721794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.749539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.798081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 350.820329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 354.652795] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:14:58 (1777043698) [ 354.776750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.799571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.840626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 354.861549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 358.990131] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:15:02 (1777043702) [ 359.064803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.097123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.183764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 359.218470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 363.231493] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:15:07 (1777043707) [ 363.380408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.415404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.470329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 363.500812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 367.858700] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:15:11 (1777043711) [ 367.914185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.944907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.998753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 368.049884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 368.063450] LNetError: 25741:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 368.089242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 368.116197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 372.040780] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:15:15 (1777043715) [ 372.108744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.156974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.240751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.llel3yRN3P [ 372.308732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.llel3yRN3P [ 372.358966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.llel3yRN3P [ 376.209886] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:15:20 (1777043720) [ 376.310385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.349487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.404764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 376.424161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 376.468843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 380.087117] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:15:24 (1777043724) [ 380.221958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.248129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.314680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 381.647740] LNet: Message forwarding will be enabled in 2s [ 384.735162] LNet: Message forwarding enabled [ 384.991145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 385.013451] LNet: Message forwarding will stop in 2s to 6s [ 389.366893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 389.821625] LNet: Message forwarding disabled [ 390.970987] LNet: Message forwarding will be enabled in 2s [ 394.779714] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:15:38 (1777043738) [ 394.810264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.560236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.621318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 399.470697] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:15:43 (1777043743) [ 399.564785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.587751] LNet: 28764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.595667] LNet: 28764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 399.620481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.649637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 399.668258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 399.693977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 399.712633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 399.735565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 399.760803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 399.785709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 399.807765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 399.826538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 399.857644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 399.891678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 399.918571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 399.952612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 399.980532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 400.005436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 400.023613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 400.048715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 400.079312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 400.101502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 400.119847] LNetError: 28808:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 404.305278] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:15:48 (1777043748) [ 404.357910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.396120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.477469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.246665] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:15:52 (1777043752) [ 409.368283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg250-server [ 410.376288] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:15:54 (1777043754) [ 410.463586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.488865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.521409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 410.534323] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 411.407340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[51-55]@tcp [ 411.417934] LNet: 30523:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 411.447337] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.52@tcp is being used as a gateway but routing feature is not turned on [ 411.461854] LNetError: 30313:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.52@tcp has gone from up to down [ 412.455867] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 412.464027] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 413.483026] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.55@tcp is being used as a gateway but routing feature is not turned on [ 413.492161] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 414.502275] LNetError: 30525:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.54@tcp has gone from up to down [ 414.513230] LNetError: 30525:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 415.521230] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:15:59 (1777043759) [ 415.588991] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.53@tcp is being used as a gateway but routing feature is not turned on [ 415.598075] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 416.609454] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.51@tcp [ 416.732900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg250-server [ 417.920633] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:16:01 (1777043761) [ 418.049911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.090190] LNetError: 31455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 418.108563] LNet: Removed LNI 192.168.202.50@tcp [ 418.153558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.202915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 418.224100] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 418.711819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[51-56/2]@tcp [ 418.726878] LNet: 31575:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 418.732858] LNet: 31575:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 418.739791] LNetError: 31469:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.51@tcp has gone from up to down [ 418.747534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[51-56/2]@tcp [ 422.903737] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:16:06 (1777043766) [ 423.081567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.097914] LNetError: 32166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.109166] LNet: Removed LNI 192.168.202.50@tcp [ 423.125440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.188326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 423.232309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.260229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.307460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.338593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.362855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.404111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 423.463207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.501376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.560649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.607132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.630291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.667433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 423.732019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.752875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.793113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.839718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.861191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.888895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 423.929405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.953955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.000932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.071946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.095327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.127467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 424.189374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.212803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.258276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.328408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.355289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.385964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 424.420536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.446840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.491756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.580856] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:16:12 (1777043772) [ 429.784864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg250-server [ 430.958694] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:16:14 (1777043774) [ 432.134454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg250-server [ 433.395325] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:16:17 (1777043777) [ 434.011181] Key type .llcrypt unregistered [ 434.013685] Key type ._llcrypt unregistered [ 438.578074] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 438.711958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 438.714428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 438.719816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 441.081203] Key type ._llcrypt registered [ 441.083162] Key type .llcrypt registered [ 441.196611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 441.205975] alg: No test for adler32 (adler32-zlib) [ 442.062298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.089836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 442.104934] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 442.599993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 442.615841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 444.026814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 444.041600] LNet: Removed LNI 10.1.2.3@tcp [ 444.112789] LNet: 34286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.117410] LNetError: 34286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 444.128894] LNet: Removed LNI 192.168.202.50@tcp [ 444.431692] Key type .llcrypt unregistered [ 444.433188] Key type ._llcrypt unregistered [ 450.386136] Key type ._llcrypt registered [ 450.388711] Key type .llcrypt registered [ 451.046441] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:16:35 (1777043795) [ 451.357270] Key type .llcrypt unregistered [ 451.359401] Key type ._llcrypt unregistered [ 454.639802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 454.683197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 454.692495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 454.701092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 457.069308] Key type ._llcrypt registered [ 457.070833] Key type .llcrypt registered [ 457.166239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 457.174743] alg: No test for adler32 (adler32-zlib) [ 458.047981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.084896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 458.100669] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 458.521931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 458.542762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 459.922851] LNet: 35778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.929163] LNetError: 35778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.935464] LNet: Removed LNI 192.168.202.50@tcp [ 459.946639] LNet: Removed LNI 10.1.2.3@tcp [ 460.317708] Key type .llcrypt unregistered [ 460.319703] Key type ._llcrypt unregistered [ 466.280157] Key type ._llcrypt registered [ 466.281851] Key type .llcrypt registered [ 467.007749] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:16:51 (1777043811) [ 467.265821] Key type .llcrypt unregistered [ 467.267141] Key type ._llcrypt unregistered [ 470.111064] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 470.198268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 470.205148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 470.207616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 472.529490] Key type ._llcrypt registered [ 472.530976] Key type .llcrypt registered [ 472.624968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 472.638170] alg: No test for adler32 (adler32-zlib) [ 473.471597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.501706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 473.516768] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 474.532985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 474.545288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 475.762669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 475.779040] LNet: Removed LNI 10.1.2.3@tcp [ 475.790084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 475.817066] LNet: Removed LNI 10.1.2.31@tcp [ 475.825882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 475.840068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 476.327265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 476.338221] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 476.946436] LNet: 37690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.951589] LNetError: 37690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 476.957486] LNet: Removed LNI 10.1.2.3@tcp [ 476.970191] LNet: Removed LNI 10.1.2.31@tcp [ 477.358443] Key type .llcrypt unregistered [ 477.361839] Key type ._llcrypt unregistered [ 483.471905] Key type ._llcrypt registered [ 483.473561] Key type .llcrypt registered [ 484.329486] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:17:08 (1777043828) [ 484.538969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.550441] alg: No test for adler32 (adler32-zlib) [ 485.389185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.419705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 485.438981] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 485.890275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.900628] LNet: 38641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.903866] LNetError: 38641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 485.912594] LNet: Removed LNI 192.168.202.50@tcp [ 485.924345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.964023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.50@tcp [ 485.977745] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 486.360354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.50@tcp [ 486.385899] LNet: Removed LNI 192.168.202.50@tcp [ 486.434350] LNet: 38776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.779476] Key type .llcrypt unregistered [ 486.782090] Key type ._llcrypt unregistered [ 492.450553] Key type ._llcrypt registered [ 492.451600] Key type .llcrypt registered [ 493.111832] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:17:17 (1777043837) [ 494.291671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg250-server [ 495.348880] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:17:19 (1777043839) [ 495.775395] Key type .llcrypt unregistered [ 495.776940] Key type ._llcrypt unregistered [ 499.151727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 499.207301] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 499.209966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 499.215675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 501.589425] Key type ._llcrypt registered [ 501.590923] Key type .llcrypt registered [ 501.700422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 501.712304] alg: No test for adler32 (adler32-zlib) [ 502.553921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.586672] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 502.983348] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 504.275138] LNet: 40608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.284276] LNetError: 40608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.287984] LNet: Removed LNI 192.168.202.50@tcp1 [ 504.293792] LNet: Removed LNI 10.1.2.3@tcp2 [ 504.656633] Key type .llcrypt unregistered [ 504.660776] Key type ._llcrypt unregistered [ 510.910987] Key type ._llcrypt registered [ 510.912825] Key type .llcrypt registered [ 511.636637] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:17:35 (1777043855) [ 511.999827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 512.010092] alg: No test for adler32 (adler32-zlib) [ 512.841136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.871297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 512.885314] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 513.339957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.352830] LNet: 41577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.355856] LNetError: 41577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.367517] LNet: Removed LNI 192.168.202.50@tcp [ 513.381638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.416768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 513.428313] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 517.527880] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:17:41 (1777043861) [ 517.626663] LNet: 42165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.637745] LNetError: 42165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.660737] LNet: Removed LNI 192.168.202.50@tcp [ 518.002590] Key type .llcrypt unregistered [ 518.004740] Key type ._llcrypt unregistered [ 521.566231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 521.648506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 521.656225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 521.662960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 524.068152] Key type ._llcrypt registered [ 524.070161] Key type .llcrypt registered [ 524.196530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 524.206320] alg: No test for adler32 (adler32-zlib) [ 525.046773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.093759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.121974] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 525.566541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 525.581552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 526.890355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.907947] LNet: 42911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.913520] LNetError: 42911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.917526] LNet: Removed LNI 192.168.202.50@tcp [ 526.924813] LNet: Removed LNI 10.1.2.3@tcp [ 526.944380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.010913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 527.031445] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 527.036955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 527.307872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.319679] LNet: Removed LNI 192.168.202.50@tcp [ 527.326472] LNet: Removed LNI 10.1.2.3@tcp [ 527.339632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.376540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 527.392069] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 527.396979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 527.597104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 527.607711] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 528.133527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.150687] LNet: 43100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.154994] LNet: 43100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 528.160941] LNetError: 43100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.167756] LNetError: 43100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 528.172455] LNet: Removed LNI 192.168.202.50@tcp [ 528.174185] LNet: Removed LNI 10.1.2.3@tcp [ 528.185539] LNet: Removed LNI 192.168.202.50@tcp2 [ 528.200806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.241798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 528.258635] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 528.265428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 528.270356] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 532.076988] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:17:56 (1777043876) [ 532.189906] LNet: 43694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.196962] LNetError: 43694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.205923] LNetError: 43694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 532.225048] LNet: Removed LNI 192.168.202.50@tcp [ 532.232661] LNet: Removed LNI 10.1.2.3@tcp [ 532.247860] LNet: Removed LNI 192.168.202.50@tcp2 [ 532.574641] Key type .llcrypt unregistered [ 532.577104] Key type ._llcrypt unregistered [ 536.276900] Key type ._llcrypt registered [ 536.279056] Key type .llcrypt registered [ 536.467777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.482433] alg: No test for adler32 (adler32-zlib) [ 537.350989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.409765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 537.433063] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 538.097968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.108976] LNet: 44093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.113950] LNetError: 44093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.128625] LNet: Removed LNI 192.168.202.50@tcp [ 538.143978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.190717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 538.208880] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 538.543674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.564686] LNet: Removed LNI 192.168.202.50@tcp [ 538.582172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.629865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 538.646764] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 538.936561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.951679] LNet: 44153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.955859] LNet: 44153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 538.959395] LNetError: 44153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.963514] LNetError: 44153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 538.980803] LNet: Removed LNI 192.168.202.50@tcp [ 538.996285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.042259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 539.062877] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 539.372751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 539.469988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 539.474781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 539.477123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 541.699498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.709093] LNet: 44231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.714338] LNetError: 44231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.729257] LNet: Removed LNI 192.168.202.50@tcp [ 541.749917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.791914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 541.810897] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 545.688208] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:18:09 (1777043889) [ 545.792632] LNet: 44828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.796182] LNetError: 44828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.804713] LNet: Removed LNI 192.168.202.50@tcp [ 546.077973] Key type .llcrypt unregistered [ 546.079403] Key type ._llcrypt unregistered [ 549.402858] Key type ._llcrypt registered [ 549.407173] Key type .llcrypt registered [ 549.566832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 549.586377] alg: No test for adler32 (adler32-zlib) [ 550.461967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.487912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.516205] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 551.141964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.163610] LNet: 45225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.169348] LNetError: 45225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.181978] LNet: Removed LNI 192.168.202.50@tcp [ 551.197183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.250981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 551.269859] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 555.091555] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:18:19 (1777043899) [ 555.272607] LNet: 45817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.279187] LNetError: 45817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.299914] LNet: Removed LNI 192.168.202.50@tcp [ 555.647602] Key type .llcrypt unregistered [ 555.648994] Key type ._llcrypt unregistered [ 558.855764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 558.979896] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 558.985250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 558.987934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 561.289827] Key type ._llcrypt registered [ 561.292242] Key type .llcrypt registered [ 561.393981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 561.402537] alg: No test for adler32 (adler32-zlib) [ 562.269968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.308867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 562.331595] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 562.940953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.953825] LNet: 46264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.959538] LNetError: 46264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.973630] LNet: Removed LNI 192.168.202.50@tcp [ 562.988245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.040836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 563.067639] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 567.797673] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:18:31 (1777043911) [ 568.004297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.019275] LNet: 46862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.025115] LNetError: 46862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.038358] LNet: Removed LNI 192.168.202.50@tcp [ 568.055818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.143485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 572.344192] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:18:36 (1777043916) [ 572.484380] LNet: 47444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.826772] Key type .llcrypt unregistered [ 572.828956] Key type ._llcrypt unregistered [ 576.704408] Key type ._llcrypt registered [ 576.705432] Key type .llcrypt registered [ 576.817962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 576.826378] alg: No test for adler32 (adler32-zlib) [ 577.709734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.756735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 577.773578] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 578.465780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.480795] LNet: 47841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.486385] LNetError: 47841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.514583] LNet: Removed LNI 192.168.202.50@tcp [ 578.532539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.580456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 578.599389] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 583.034624] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:18:46 (1777043926) [ 583.100441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.120626] LNet: 48428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.126256] LNetError: 48428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.153205] LNet: Removed LNI 192.168.202.50@tcp [ 583.176554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.230556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 586.927536] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:18:50 (1777043930) [ 587.955477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 589.057975] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:18:53 (1777043933) [ 590.126418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 591.194727] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:18:55 (1777043935) [ 592.175614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 593.194724] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:18:57 (1777043937) [ 594.258139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 595.401433] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:18:59 (1777043939) [ 596.264884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 597.257924] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:19:01 (1777043941) [ 597.323361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.341293] LNet: 50478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.357870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.406896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.422995] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 597.866539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.50 [ 597.879562] LNet: 50617:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 597.886963] LNet: Removed LNI 192.168.202.50@tcp [ 601.770239] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:19:05 (1777043945) [ 601.823769] LNet: 51178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.828641] LNetError: 51178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.840376] LNet: Removed LNI 192.168.202.50@tcp [ 602.290784] Key type .llcrypt unregistered [ 602.292422] Key type ._llcrypt unregistered [ 605.860339] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 606.057538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 606.060741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 606.064595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 608.533597] Key type ._llcrypt registered [ 608.535523] Key type .llcrypt registered [ 608.639689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.653310] alg: No test for adler32 (adler32-zlib) [ 609.776791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 613.807921] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:19:17 (1777043957) [ 613.875679] LNet: 52086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.882621] LNetError: 52086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 613.905769] LNet: Removed LNI 10.1.2.3@tcp [ 614.350455] Key type .llcrypt unregistered [ 614.352676] Key type ._llcrypt unregistered [ 617.826827] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 617.939781] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 617.942579] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 617.946679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 620.360580] Key type ._llcrypt registered [ 620.362335] Key type .llcrypt registered [ 620.470268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.477538] alg: No test for adler32 (adler32-zlib) [ 621.365068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 625.744860] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:19:29 (1777043969) [ 625.871317] LNet: 52981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.879952] LNetError: 52981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 625.907099] LNet: Removed LNI 10.1.2.3@tcp [ 626.380474] Key type .llcrypt unregistered [ 626.384332] Key type ._llcrypt unregistered [ 630.369403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 630.554815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 630.559150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 630.563642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 633.048138] Key type ._llcrypt registered [ 633.049717] Key type .llcrypt registered [ 633.158445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 633.172460] alg: No test for adler32 (adler32-zlib) [ 634.057276] LNet: 53286:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 634.068201] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 637.821605] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:19:41 (1777043981) [ 637.973738] LNet: 53878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.980627] LNetError: 53878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 637.993812] LNet: Removed LNI 10.1.2.3@tcp [ 638.394562] Key type .llcrypt unregistered [ 638.396167] Key type ._llcrypt unregistered [ 641.570773] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 641.747108] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 641.753364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 641.758214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 644.241612] Key type ._llcrypt registered [ 644.243884] Key type .llcrypt registered [ 644.364748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 644.378624] alg: No test for adler32 (adler32-zlib) [ 645.242336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.283790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.744711] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:19:54 (1777043994) [ 651.000549] LNet: 55107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.008540] LNetError: 55107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 651.019378] LNet: Removed LNI 10.1.2.3@tcp [ 651.414801] Key type .llcrypt unregistered [ 651.417270] Key type ._llcrypt unregistered [ 654.257793] Key type ._llcrypt registered [ 654.259596] Key type .llcrypt registered [ 654.344237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.355905] alg: No test for adler32 (adler32-zlib) [ 655.177862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 655.198239] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 659.337314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 659.801430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_interrupt -r 1 [ 659.822223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_interrupt -r 1 [ 659.844341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 659.856154] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 660.008688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 660.033626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 660.064641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 660.090310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 660.111603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 660.136952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 660.153277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.178292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 660.187784] LNet: 55544:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 660.211570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 660.240313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.273088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 660.300260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 660.319289] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 660.329811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.357606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 660.382392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 660.713453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_dropped -r 1 [ 660.735078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_dropped -r 1 [ 660.752618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 660.763389] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 660.766789] LNet: Skipped 1 previous similar message [ 660.864823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 660.884522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 660.898880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 660.916644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 660.931651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 660.953458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 660.971185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.988907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 660.999673] LNet: 55653:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 661.013609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.034856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.051092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.068645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.085049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.103468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.128085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 661.373302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_aborted -r 1 [ 661.394688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_aborted -r 1 [ 661.416661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 661.557613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 661.575912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 661.593829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 661.617988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 661.638660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 661.655239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 661.671678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 661.688122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 661.714420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.733631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.748602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.768642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.783820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.801269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.817700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 662.091885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_no_route -r 1 [ 662.111550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_no_route -r 1 [ 662.133430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 662.146671] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 662.150332] LNet: Skipped 1 previous similar message [ 662.295169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 662.323513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 662.339756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 662.355607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 662.376150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 662.395874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 662.411643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 662.431564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 662.444136] LNet: 55871:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 662.450119] LNet: 55871:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 662.463894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 662.489657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 662.515581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 662.540232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 662.563539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 662.586930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 662.604260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 662.880836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_timeout -r 1 [ 662.897240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_timeout -r 1 [ 662.919205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 663.069988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 663.095588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 663.125609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 663.151629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 663.175888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 663.201506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 663.222086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 663.245137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 663.262306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 663.288769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 663.308122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 663.337337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 663.355505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 663.376852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 663.398627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 663.700908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_error -r 1 [ 663.729622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_error -r 1 [ 663.758668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 663.913362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 663.931829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 663.955342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 663.981681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 664.008587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 664.036056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 664.057728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 664.082695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 664.117098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 664.143635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 664.166678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 664.190720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 664.215541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 664.232334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 664.251081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 664.404280] LNet: 56131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.409599] LNetError: 56131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.456024] LNet: Removed LNI 192.168.202.50@tcp [ 665.682733] Key type .llcrypt unregistered [ 665.685788] Key type ._llcrypt unregistered [ 674.721244] Key type ._llcrypt registered [ 674.724114] Key type .llcrypt registered [ 675.510649] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:20:19 (1777044019) [ 676.122806] Key type .llcrypt unregistered [ 676.125335] Key type ._llcrypt unregistered [ 679.832068] Key type ._llcrypt registered [ 679.833700] Key type .llcrypt registered [ 679.936938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.956144] alg: No test for adler32 (adler32-zlib) [ 680.793253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 680.817125] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 684.899828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 689.378409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.399630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 689.413051] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 690.121451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_interrupt -r 1 [ 690.147365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_interrupt -r 1 [ 690.165299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_interrupt -r 1 [ 690.194657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_interrupt -r 1 [ 690.224364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_interrupt -r 1 [ 690.249636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_interrupt -r 1 [ 690.272164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_interrupt -r 1 [ 690.293651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_interrupt -r 1 [ 690.318880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 690.327983] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 690.330622] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 690.442456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 690.464742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 690.481064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 690.497553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 690.514489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 690.532490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 690.552962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 690.571205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 690.580797] LNet: 57596:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 690.593592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 690.617611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 690.638984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 690.660810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 690.684399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 690.709360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 690.729826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 690.981225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_dropped -r 1 [ 691.008279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_dropped -r 1 [ 691.032885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_dropped -r 1 [ 691.052611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_dropped -r 1 [ 691.076667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_dropped -r 1 [ 691.099455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_dropped -r 1 [ 691.126479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_dropped -r 1 [ 691.141800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_dropped -r 1 [ 691.170707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 691.181044] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 691.184481] LNet: Skipped 2 previous similar messages [ 691.306587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 691.329255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 691.346907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 691.363666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 691.381564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 691.401689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 691.424448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 691.443372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 691.455712] LNet: 57717:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 691.464751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 691.479333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 691.498692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 691.515825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 691.537770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 691.556892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 691.573673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 691.823566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_aborted -r 1 [ 691.842290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_aborted -r 1 [ 691.861353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_aborted -r 1 [ 691.881350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_aborted -r 1 [ 691.897341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_aborted -r 1 [ 691.914760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_aborted -r 1 [ 691.935792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_aborted -r 1 [ 691.955968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_aborted -r 1 [ 691.981641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 692.087285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 692.104433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 692.120955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 692.136587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 692.152760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 692.172355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 692.200801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 692.224436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 692.250875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 692.281313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 692.302479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 692.323206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 692.345490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 692.362864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 692.381747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 692.705518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_no_route -r 1 [ 692.726487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_no_route -r 1 [ 692.747081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_no_route -r 1 [ 692.770987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_no_route -r 1 [ 692.802498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_no_route -r 1 [ 692.831715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_no_route -r 1 [ 692.855379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_no_route -r 1 [ 692.876656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_no_route -r 1 [ 692.896855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 692.908436] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 692.912224] LNet: Skipped 6 previous similar messages [ 693.026786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 693.044501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 693.068803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 693.087226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 693.115667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 693.135897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 693.157656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 693.189594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 693.203764] LNet: 57959:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 693.207492] LNet: 57959:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 693.216881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 693.233101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 693.253663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 693.272671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 693.294702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 693.314587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 693.335427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 693.595946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_timeout -r 1 [ 693.620443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_timeout -r 1 [ 693.647251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_timeout -r 1 [ 693.671587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_timeout -r 1 [ 693.687885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_timeout -r 1 [ 693.713464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_timeout -r 1 [ 693.735686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_timeout -r 1 [ 693.757581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_timeout -r 1 [ 693.788603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 693.929383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 693.950079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 693.966298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 693.982726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 694.002295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 694.023344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 694.037661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 694.054454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 694.073290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 694.091980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 694.113601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 694.133034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 694.150792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 694.172058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 694.188170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 694.501429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_error -r 1 [ 694.515425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_error -r 1 [ 694.532274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_error -r 1 [ 694.560550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_error -r 1 [ 694.590913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_error -r 1 [ 694.622609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_error -r 1 [ 694.644615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_error -r 1 [ 694.667514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_error -r 1 [ 694.692922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 694.825786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 694.841636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 694.863180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 694.879167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 694.898126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 694.913614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 694.933511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 694.953605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 694.973244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 694.989378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 695.007888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 695.025332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 695.048922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 695.072329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 695.091304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 697.898663] LNet: 58277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.903507] LNetError: 58277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.908381] LNet: Removed LNI 192.168.202.50@tcp [ 698.923417] LNet: Removed LNI 192.168.202.50@tcp1 [ 699.157810] Key type .llcrypt unregistered [ 699.159782] Key type ._llcrypt unregistered [ 707.311121] Key type ._llcrypt registered [ 707.312287] Key type .llcrypt registered [ 708.055588] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:20:52 (1777044052) [ 708.434320] Key type .llcrypt unregistered [ 708.436845] Key type ._llcrypt unregistered [ 710.981811] Key type ._llcrypt registered [ 710.983389] Key type .llcrypt registered [ 711.077486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.088747] alg: No test for adler32 (adler32-zlib) [ 711.911973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 711.946539] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 716.185950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 716.522802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_dropped -r 1 [ 716.558851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_dropped -r 1 [ 716.577610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 716.588666] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 716.722757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 716.742783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 716.760103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 716.777462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 716.802696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 716.827415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 716.851568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 716.871546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 716.881906] LNet: 59531:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 716.891553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 716.910439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 716.926398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 716.942337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 716.959967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 716.983428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 717.012983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 717.272358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_error -r 1 [ 717.291748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_error -r 1 [ 717.310369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 717.322557] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 717.413132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 717.440817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 717.462738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 717.481942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 717.498259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 717.516563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 717.531694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 717.546702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 717.555567] LNet: 59640:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 717.565319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 717.580758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 717.597949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 717.615159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.631904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 717.657218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 717.682277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 717.928591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_timeout -r 1 [ 717.957579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_timeout -r 1 [ 717.978550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 718.090345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 718.106075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 718.120413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 718.137610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 718.157370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 718.176412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 718.199910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 718.216121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 718.231344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 718.246627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.267409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 718.287141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 718.307086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 718.328857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 718.349160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 718.485893] LNet: 59790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.491868] LNetError: 59790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.516341] LNet: Removed LNI 192.168.202.50@tcp [ 718.713375] Key type .llcrypt unregistered [ 718.715651] Key type ._llcrypt unregistered [ 727.195922] Key type ._llcrypt registered [ 727.197608] Key type .llcrypt registered [ 727.815231] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:21:11 (1777044071) [ 728.241407] Key type .llcrypt unregistered [ 728.242814] Key type ._llcrypt unregistered [ 731.375344] Key type ._llcrypt registered [ 731.377931] Key type .llcrypt registered [ 731.524408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.539767] alg: No test for adler32 (adler32-zlib) [ 732.367737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 732.385346] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 737.007754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 741.551972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.578720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 741.596687] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 742.532379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_dropped -r 1 [ 742.551694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e remote_dropped -r 1 [ 742.569987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_dropped -r 1 [ 742.593850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e remote_dropped -r 1 [ 742.615898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e remote_dropped -r 1 [ 742.637663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e remote_dropped -r 1 [ 742.669568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e remote_dropped -r 1 [ 742.704528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e remote_dropped -r 1 [ 742.730104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 742.745747] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 742.754291] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.150@tcp [ 742.913194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 742.934317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 742.957797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 742.979666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 743.002623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 743.033576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 743.059288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 743.077979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 743.086626] LNet: 61256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 743.095692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 743.117126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 743.150572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 743.183418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 743.211723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 743.237725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 743.259547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 743.281344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 743.591227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_error -r 1 [ 743.622277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e remote_error -r 1 [ 743.655091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_error -r 1 [ 743.675987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e remote_error -r 1 [ 743.699235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e remote_error -r 1 [ 743.727058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e remote_error -r 1 [ 743.754736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e remote_error -r 1 [ 743.778737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e remote_error -r 1 [ 743.802102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 743.814119] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 743.818731] LNet: Skipped 2 previous similar messages [ 743.946408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 743.965814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 743.988158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 744.015666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 744.038282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 744.059543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 744.076911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 744.091608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 744.100577] LNet: 61379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 744.115752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 744.138414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 744.164431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 744.192388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 744.208437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 744.227586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 744.247131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 744.265504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 744.624442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_timeout -r 1 [ 744.648888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e remote_timeout -r 1 [ 744.683950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_timeout -r 1 [ 744.712930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e remote_timeout -r 1 [ 744.749632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e remote_timeout -r 1 [ 744.779305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e remote_timeout -r 1 [ 744.810136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e remote_timeout -r 1 [ 744.836944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e remote_timeout -r 1 [ 744.868947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.150@tcp [ 744.889736] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 745.012794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 745.028929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 745.048594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 745.066876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 745.083881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 745.100291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 745.115926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 745.132748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 745.143783] LNet: 61502:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 745.156892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 745.174611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 745.191621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 745.216047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 745.241059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 745.258264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 745.275846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 745.294761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 747.990506] LNet: 61579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.993318] LNetError: 61579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.996301] LNet: Removed LNI 192.168.202.50@tcp [ 748.002841] LNet: Removed LNI 192.168.202.50@tcp1 [ 748.191691] Key type .llcrypt unregistered [ 748.193347] Key type ._llcrypt unregistered [ 756.019313] Key type ._llcrypt registered [ 756.020743] Key type .llcrypt registered [ 756.711365] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:21:40 (1777044100) [ 757.002527] Key type .llcrypt unregistered [ 757.004868] Key type ._llcrypt unregistered [ 759.890295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.022052] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.024089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.028639] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.374720] Key type ._llcrypt registered [ 762.376800] Key type .llcrypt registered [ 762.467677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.477062] alg: No test for adler32 (adler32-zlib) [ 763.305528] LNet: 62655:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 763.311145] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 763.328085] LNet: 62669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.332647] LNetError: 62669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.344324] LNet: Removed LNI 192.168.202.50@tcp [ 763.646415] Key type .llcrypt unregistered [ 763.648786] Key type ._llcrypt unregistered [ 766.190821] Key type ._llcrypt registered [ 766.192258] Key type .llcrypt registered [ 766.272041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.282608] alg: No test for adler32 (adler32-zlib) [ 767.129583] LNet: 62927:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 767.139834] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 767.141418] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 767.162774] LNet: 62941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.170952] LNetError: 62941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.174666] LNet: Removed LNI 192.168.202.50@tcp [ 767.184169] LNet: Removed LNI 10.1.2.3@tcp1 [ 767.515383] Key type .llcrypt unregistered [ 767.516798] Key type ._llcrypt unregistered [ 770.211270] Key type ._llcrypt registered [ 770.213200] Key type .llcrypt registered [ 770.313169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.321833] alg: No test for adler32 (adler32-zlib) [ 771.157277] LNet: 63200:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 771.166841] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 771.168862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.193512] LNet: 63214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.198632] LNetError: 63214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.201029] LNet: Removed LNI 192.168.202.50@tcp [ 771.208632] LNet: Removed LNI 10.1.2.3@tcp [ 771.557369] Key type .llcrypt unregistered [ 771.558948] Key type ._llcrypt unregistered [ 774.255731] Key type ._llcrypt registered [ 774.256896] Key type .llcrypt registered [ 774.336541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.342775] alg: No test for adler32 (adler32-zlib) [ 775.187581] LNet: 63473:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 775.194315] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 775.197425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.228608] LNet: 63487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.233421] LNetError: 63487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.237115] LNet: Removed LNI 192.168.202.50@tcp [ 775.244831] LNet: Removed LNI 10.1.2.3@tcp [ 775.545626] Key type .llcrypt unregistered [ 775.547531] Key type ._llcrypt unregistered [ 778.355597] Key type ._llcrypt registered [ 778.357260] Key type .llcrypt registered [ 778.455151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.471884] alg: No test for adler32 (adler32-zlib) [ 779.300084] LNet: 63751:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 779.305996] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 779.308194] LNetError: 63751:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 779.311543] LNetError: Error -100 starting up LNI tcp [ 779.319568] LNet: Removed LNI 192.168.202.50@tcp [ 779.701620] Key type .llcrypt unregistered [ 779.704563] Key type ._llcrypt unregistered [ 785.524818] Key type ._llcrypt registered [ 785.526540] Key type .llcrypt registered [ 786.265332] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:22:10 (1777044130) [ 786.751619] Key type .llcrypt unregistered [ 786.753513] Key type ._llcrypt unregistered [ 789.610388] Key type ._llcrypt registered [ 789.611868] Key type .llcrypt registered [ 789.716353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.725458] alg: No test for adler32 (adler32-zlib) [ 790.552252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 790.569090] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 794.167312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 794.488834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e network_timeout -r 1 [ 794.507563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e network_timeout -r 1 [ 794.528251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 794.545666] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 794.647641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 794.655136] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 794.662630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 794.676535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 794.694519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 794.712329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 794.732322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 794.749250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 794.766790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 794.776848] LNet: 65004:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 794.786164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 794.801888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 794.823714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 794.849081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 794.867319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 794.887928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 795.035261] LNet: 65043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.038144] LNetError: 65043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.055578] LNet: Removed LNI 192.168.202.50@tcp [ 795.222597] Key type .llcrypt unregistered [ 795.224832] Key type ._llcrypt unregistered [ 802.598679] Key type ._llcrypt registered [ 802.599846] Key type .llcrypt registered [ 802.747154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.757155] alg: No test for adler32 (adler32-zlib) [ 803.592708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 803.609833] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 807.321783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 811.447770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.469811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 811.482485] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 812.186992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e network_timeout -r 1 [ 812.204371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e network_timeout -r 1 [ 812.221342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e network_timeout -r 1 [ 812.240478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e network_timeout -r 1 [ 812.259207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e network_timeout -r 1 [ 812.276321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e network_timeout -r 1 [ 812.297727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e network_timeout -r 1 [ 812.320393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e network_timeout -r 1 [ 812.339697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 812.350833] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 812.443637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 812.459454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 812.474861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 812.491157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 812.505267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 812.523796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 812.541386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 812.559127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 812.568333] LNet: 65919:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 812.581028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 812.603481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 812.619195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 812.636643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 812.652680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 812.670879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 815.112321] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 815.258125] LNet: 65992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.261576] LNetError: 65992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.265112] LNet: Removed LNI 192.168.202.50@tcp [ 815.277560] LNet: Removed LNI 192.168.202.50@tcp1 [ 815.441578] Key type .llcrypt unregistered [ 815.443219] Key type ._llcrypt unregistered [ 822.010103] Key type ._llcrypt registered [ 822.011193] Key type .llcrypt registered [ 822.570454] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:22:46 (1777044166) [ 822.739187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.750271] alg: No test for adler32 (adler32-zlib) [ 823.551372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.575574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.586971] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 823.970555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 823.980492] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 824.386906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 824.417327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 824.438590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 824.457256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 824.473144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 824.488667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 824.504507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.50@tcp [ 824.515613] LNetError: 67089:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.50@tcp to recovery [ 824.607283] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 824.607318] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 826.655230] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 827.752904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 830.754622] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 830.990515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 831.006016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 831.042334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.054907] LNet: 67196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.058847] LNetError: 67196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.062551] LNet: Removed LNI 192.168.202.50@tcp [ 831.078028] LNet: Removed LNI 192.168.202.50@tcp1 [ 831.089798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.115481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.139230] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 831.488354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 831.499472] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 831.908221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 831.926240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 831.956935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 831.988381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 832.008409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 832.025707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 832.047856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 832.064979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.50@tcp [ 832.077502] LNetError: 67468:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.50@tcp to recovery [ 834.213419] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 834.218740] LNet: Skipped 1 previous similar message [ 834.306372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 838.303596] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 842.775846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 842.796677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 842.813993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 845.805641] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:23:10 (1777044190) [ 845.897125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.909080] LNet: 68184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.912866] LNetError: 68184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.916638] LNetError: 68184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.919455] LNet: Removed LNI 192.168.202.50@tcp [ 845.926852] LNet: Removed LNI 192.168.202.50@tcp1 [ 845.936264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.965819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.978095] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 846.289167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 846.301895] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 846.677555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 846.706640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 846.721207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 846.740063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 846.757296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 846.772078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 846.786247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.50@tcp --health 0 [ 846.824092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 847.007338] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.50@tcp [ 847.007626] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 847.016751] LNet: Skipped 1 previous similar message [ 861.419369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 861.490629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 861.503829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 861.518524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.50@tcp --health 500 [ 861.561551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 867.798898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 867.826401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.835856] LNet: 68676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.839579] LNetError: 68676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.842882] LNetError: 68676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 867.846448] LNet: Removed LNI 192.168.202.50@tcp [ 867.853627] LNet: Removed LNI 192.168.202.50@tcp1 [ 867.864191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.890903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.907180] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 868.265282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 868.277590] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 868.664250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 868.688609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 868.720570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 868.736926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 868.755871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 868.773317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 868.789669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 868.812487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.50@tcp --health 0 [ 869.919389] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 869.923600] LNet: Skipped 3 previous similar messages [ 871.001641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 879.451604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 879.467959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 879.485627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 883.042464] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:23:47 (1777044227) [ 883.277142] LNet: 69720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.280142] LNetError: 69720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.284251] LNetError: 69720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.288405] LNet: Removed LNI 192.168.202.50@tcp [ 883.296681] LNet: Removed LNI 192.168.202.50@tcp1 [ 883.556807] Key type .llcrypt unregistered [ 883.558339] Key type ._llcrypt unregistered [ 886.207131] Key type ._llcrypt registered [ 886.208610] Key type .llcrypt registered [ 886.285863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.294277] alg: No test for adler32 (adler32-zlib) [ 887.122394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 887.136605] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 891.216090] Lustre: DEBUG MARKER: Initial discovery [ 891.243871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.150@tcp [ 892.197704] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 892.259505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 892.277643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.150@tcp [ 892.288330] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 892.298064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 892.314479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.150@tcp -r 1 -m GET -l 3 [ 892.331725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.150@tcp -r 1 -m GET -e local_error [ 892.348350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.150@tcp -r 1 -m PUT -l 6 [ 892.870311] Lustre: DEBUG MARKER: Force oleg250-client.virtnet to discover 192.168.202.150@tcp (in background) [ 892.926060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.150@tcp [ 893.279337] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 893.417960] Lustre: DEBUG MARKER: Force oleg250-server to discover 192.168.202.50@tcp [ 894.338378] Lustre: DEBUG MARKER: Wait for 70480 [ 895.967577] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 896.513665] Lustre: DEBUG MARKER: Finished wait on 70480 [ 896.558913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 896.580282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 896.702857] LNet: 70929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.708399] LNetError: 70929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.716503] LNet: Removed LNI 192.168.202.50@tcp [ 896.880451] Key type .llcrypt unregistered [ 896.881964] Key type ._llcrypt unregistered [ 903.695227] Key type ._llcrypt registered [ 903.696743] Key type .llcrypt registered [ 904.372306] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:24:08 (1777044248) [ 904.626436] Key type .llcrypt unregistered [ 904.628123] Key type ._llcrypt unregistered [ 907.356134] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 907.439294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 907.442117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 907.444655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.674658] Key type ._llcrypt registered [ 909.675690] Key type .llcrypt registered [ 909.768457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 909.775438] alg: No test for adler32 (adler32-zlib) [ 910.623189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.666869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.682537] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 911.068306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 911.080551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 915.381206] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:24:19 (1777044259) [ 915.443931] LNet: 72993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.449755] LNetError: 72993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.455379] LNet: Removed LNI 192.168.202.50@tcp [ 915.469464] LNet: Removed LNI 10.1.2.3@tcp [ 915.784419] Key type .llcrypt unregistered [ 915.787142] Key type ._llcrypt unregistered [ 918.402175] Key type ._llcrypt registered [ 918.403392] Key type .llcrypt registered [ 918.529947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.538509] alg: No test for adler32 (adler32-zlib) [ 919.367411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.397958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.414100] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 919.731942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 919.745916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.242910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.50@tcp [ 921.312190] LNetError: Unexpected error -22 connecting to 192.168.202.50@tcp at host 192.168.202.50:988 [ 923.366124] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:24:27 (1777044267) [ 923.407821] LNet: 74122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.412109] LNetError: 74122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.415177] LNet: Removed LNI 192.168.202.50@tcp [ 923.424596] LNet: Removed LNI 10.1.2.3@tcp [ 923.701438] Key type .llcrypt unregistered [ 923.703063] Key type ._llcrypt unregistered [ 926.181944] Key type ._llcrypt registered [ 926.183397] Key type .llcrypt registered [ 926.296208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.308150] alg: No test for adler32 (adler32-zlib) [ 927.138704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.168406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 927.181531] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 927.556358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 927.567584] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 927.975984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.50@tcp1 --nid 192.168.202.50@tcp2 [ 929.682938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp1 192.168.202.50@tcp2 [ 929.693690] LNetError: 74963:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.50@tcp1 to 192.168.202.50@tcp2 - Route table may be misconfigured [ 929.704804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp2 192.168.202.50@tcp1 [ 932.374418] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:24:36 (1777044276) [ 932.528122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.544823] LNet: 75536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.549464] LNetError: 75536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.553759] LNet: Removed LNI 192.168.202.50@tcp1 [ 933.605915] LNet: Removed LNI 192.168.202.50@tcp2 [ 933.621156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.673915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.699761] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 934.115414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 934.126811] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 934.598644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 934.625577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.50@tcp [ 934.644379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp1 192.168.202.50@tcp1 [ 934.666843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.50@tcp -d 192.168.202.50@tcp -e network_timeout [ 934.691528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -e network_timeout [ 934.722286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -e network_timeout [ 934.755280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -e network_timeout [ 934.777856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 934.789043] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 934.804202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 934.853779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 934.906374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 935.711274] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 938.129938] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:24:42 (1777044282) [ 938.179132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.197153] LNet: 76391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.201881] LNetError: 76391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.205529] LNetError: 76391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 938.209910] LNet: Removed LNI 192.168.202.50@tcp [ 939.241989] LNet: Removed LNI 192.168.202.50@tcp1 [ 939.255560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.302815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 939.472699] LNet: 76434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.749724] Key type .llcrypt unregistered [ 939.752181] Key type ._llcrypt unregistered [ 945.159989] Key type ._llcrypt registered [ 945.161142] Key type .llcrypt registered [ 945.878498] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:24:49 (1777044289) [ 946.093851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.100295] alg: No test for adler32 (adler32-zlib) [ 946.929164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.663264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 948.528102] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:24:52 (1777044292) [ 948.572343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.586976] LNet: 77523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.600839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.639163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 948.661701] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 949.038467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 949.054788] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 949.514400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 949.528685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp1 [ 949.544922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp1 [ 952.339627] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:24:56 (1777044296) [ 953.192835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg250-server [ 953.968407] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:24:58 (1777044298) [ 954.926201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg250-server [ 955.755181] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:24:59 (1777044299) [ 956.573533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg250-server [ 957.321776] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:25:01 (1777044301) [ 958.090886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg250-server [ 958.825158] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:25:03 (1777044303) [ 959.596434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg250-server [ 960.331279] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:25:04 (1777044304) [ 961.102255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg250-server [ 961.811963] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:25:06 (1777044306) [ 962.642643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg250-server [ 963.382763] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:25:07 (1777044307) [ 964.127074] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg250-server [ 964.869154] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:25:09 (1777044309) [ 965.623085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg250-server [ 966.367542] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:25:10 (1777044310) [ 966.460140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.473617] LNet: 81403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.478118] LNetError: 81403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.481768] LNet: Removed LNI 192.168.202.50@tcp [ 967.525629] LNet: Removed LNI 192.168.202.50@tcp1 [ 967.534713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.560890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.572563] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 967.963746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 968.002779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 968.085641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.095611] LNet: 81550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.099451] LNetError: 81550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.102855] LNetError: 81550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 968.115439] LNet: Removed LNI 192.168.202.50@tcp [ 968.125486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.155945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 968.170491] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 968.501761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 968.567506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.641603] LNet: Removed LNI 192.168.202.50@tcp [ 969.652916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.679220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.690569] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 970.021473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 970.063324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 970.159446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.169652] LNet: 81842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.173817] LNet: 81842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 970.178353] LNetError: 81842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.182183] LNetError: 81842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 970.195605] LNet: Removed LNI 192.168.202.50@tcp [ 970.205170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.233458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 970.246474] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 970.633616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 970.737602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.813318] LNet: Removed LNI 192.168.202.50@tcp [ 971.821494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.844493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 971.855634] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 972.168450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 972.199437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 972.284590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.293714] LNetError: 82135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.296167] LNetError: 82135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 972.309749] LNet: Removed LNI 192.168.202.50@tcp [ 972.317658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.345833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 972.360585] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 972.680491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 973.818596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.887342] LNet: Removed LNI 192.168.202.50@tcp [ 974.895750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.918555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.930889] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 975.222298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 975.257820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 975.347385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.356925] LNet: 82436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.359717] LNet: 82436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 976.421783] LNet: Removed LNI 192.168.202.50@tcp [ 976.431910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.460828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 976.472503] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 976.835167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 979.057157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.067263] LNetError: 82600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.070018] LNetError: 82600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 979.079420] LNet: Removed LNI 192.168.202.50@tcp [ 979.088161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.113511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.128192] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 979.482896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 979.530571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 980.727351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.796890] LNet: Removed LNI 192.168.202.50@tcp [ 981.811313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.843912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 981.859423] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 982.177820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 983.347693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.356211] LNet: 82910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.358607] LNet: 82910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 984.421446] LNet: Removed LNI 192.168.202.50@tcp [ 984.430113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.453805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 984.469357] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 984.822913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 984.860499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 984.967783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.021628] LNet: Removed LNI 192.168.202.50@tcp [ 986.039483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.081821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 986.101723] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 986.492275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 987.660407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.671407] LNetError: 83212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.675318] LNetError: 83212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 987.684355] LNet: Removed LNI 192.168.202.50@tcp [ 987.694950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.723318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.737980] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 988.044609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 988.073890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.090161] LNet: Removed LNI 192.168.202.50@tcp [ 988.100201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.128738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.143575] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 988.481084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 991.764450] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:25:35 (1777044335) [ 991.814931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.835778] LNet: Removed LNI 192.168.202.50@tcp [ 991.848874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.897472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 991.914815] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/180] [ 992.526281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.545497] LNet: Removed LNI 192.168.202.50@tcp231 [ 992.555918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.591918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 992.605254] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/180] [ 992.609672] LNet: 84194:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.805295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 992.820685] LNet: Removed LNI 192.168.202.50@tcp231 [ 992.827438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 992.837278] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/180] [ 993.408690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.424784] LNet: Removed LNI 192.168.202.50@tcp231 [ 993.439871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.488740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 993.508525] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/0] [ 993.516496] LNet: 84346:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 993.727272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 993.751422] LNet: Removed LNI 192.168.202.50@tcp231 [ 993.773592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 993.791669] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/0] [ 994.288253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.310561] LNet: Removed LNI 192.168.202.50@tcp231 [ 994.329374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.358718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 994.370614] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/60] [ 994.537293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 994.552879] LNet: Removed LNI 192.168.202.50@tcp231 [ 994.560845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 994.572676] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/60] [ 995.166711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.187905] LNet: Removed LNI 192.168.202.50@tcp231 [ 995.200828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.231931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 995.246721] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/180] [ 995.249816] LNet: 84648:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 995.253228] LNet: 84648:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 995.437973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 995.464607] LNet: Removed LNI 192.168.202.50@tcp231 [ 995.472614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 995.487510] LNet: Added LNI 192.168.202.50@tcp231 [8/256/0/180] [ 998.773243] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:25:42 (1777044342) [ 998.870289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.889079] LNet: Removed LNI 192.168.202.50@tcp231 [ 998.903907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.937374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 998.951062] LNet: Added LNI 192.168.202.50@tcp232 [8/256/0/180] [ 999.284894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 999.302877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 999.330601] LNet: Removed LNI 192.168.202.50@tcp232 [ 999.339021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 999.350860] LNet: Added LNI 192.168.202.50@tcp232 [8/256/0/180] [ 999.673611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 999.690312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 999.708515] LNet: Removed LNI 192.168.202.50@tcp232 [ 999.717241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 999.728947] LNet: Added LNI 192.168.202.50@tcp232 [8/256/0/180] [ 1000.033263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1000.054801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1000.070772] LNet: Removed LNI 192.168.202.50@tcp232 [ 1000.078479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1000.096042] LNet: Added LNI 192.168.202.50@tcp232 [8/256/0/180] [ 1000.473826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1000.506448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1000.533812] LNet: Removed LNI 192.168.202.50@tcp232 [ 1000.544643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1003.284606] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:25:47 (1777044347) [ 1003.494665] LNet: 86484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.498289] LNet: 86484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1003.782831] Key type .llcrypt unregistered [ 1003.784698] Key type ._llcrypt unregistered [ 1006.208341] Key type ._llcrypt registered [ 1006.209805] Key type .llcrypt registered [ 1006.285595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.296239] alg: No test for adler32 (adler32-zlib) [ 1007.112509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1007.125922] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1010.652272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 1014.194741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.218518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1014.236331] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 1015.053293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_interrupt -r 2 [ 1015.072775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_interrupt -r 2 [ 1015.092493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_interrupt -r 2 [ 1015.113417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_interrupt -r 2 [ 1015.134760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_interrupt -r 2 [ 1015.165756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_interrupt -r 2 [ 1015.183989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_interrupt -r 2 [ 1015.202868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_interrupt -r 2 [ 1015.229114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1015.240235] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 1015.244328] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 1015.361776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.378379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.395586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.411333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.425780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.439863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.461338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.481358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.490685] LNet: 87144:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.500420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.518596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.535897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.557799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.577765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.595871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.837484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_dropped -r 2 [ 1015.855792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_dropped -r 2 [ 1015.882119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_dropped -r 2 [ 1015.903304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_dropped -r 2 [ 1015.920788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_dropped -r 2 [ 1015.939926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_dropped -r 2 [ 1015.962751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_dropped -r 2 [ 1015.980090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_dropped -r 2 [ 1015.997794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1016.007850] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 1016.115440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.133073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1016.146484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.163905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.182656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.199207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.214875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.229969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.241582] LNet: 87264:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.260260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.278713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.293488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.311679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.335836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.362523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.584169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_aborted -r 2 [ 1016.600726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_aborted -r 2 [ 1016.619559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_aborted -r 2 [ 1016.638197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_aborted -r 2 [ 1016.654994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_aborted -r 2 [ 1016.673343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_aborted -r 2 [ 1016.692460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_aborted -r 2 [ 1016.713189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_aborted -r 2 [ 1016.733532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1016.864393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.878695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1016.894339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.911638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.929625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.947693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.965570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.981309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.997240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.010308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.025896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.042191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.062977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.079373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.310545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_no_route -r 2 [ 1017.330287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_no_route -r 2 [ 1017.351686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_no_route -r 2 [ 1017.370676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_no_route -r 2 [ 1017.393075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_no_route -r 2 [ 1017.412917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_no_route -r 2 [ 1017.429219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_no_route -r 2 [ 1017.447557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_no_route -r 2 [ 1017.467810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1017.480156] LNet: There was an unexpected network error while writing to 192.168.202.150: rc = -22 [ 1017.484256] LNet: Skipped 2 previous similar messages [ 1017.596854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.622520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.642740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.660917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.677397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.698864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.716448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.732724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.744350] LNet: 87505:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.746442] LNet: 87505:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1017.759609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.782638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.803508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.823784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.841235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.858174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.079654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e local_timeout -r 2 [ 1018.097304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e local_timeout -r 2 [ 1018.114070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e local_timeout -r 2 [ 1018.132649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e local_timeout -r 2 [ 1018.150068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e local_timeout -r 2 [ 1018.169767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e local_timeout -r 2 [ 1018.186667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e local_timeout -r 2 [ 1018.205851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e local_timeout -r 2 [ 1018.225256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1018.335204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.349741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1018.365294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.380658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.394373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.408116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.424343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.437352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.453419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.473191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.488190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.503700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.518536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.532599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.743905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp -m GET -e remote_dropped -r 2 [ 1018.761587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.150@tcp1 -m GET -e remote_dropped -r 2 [ 1018.777376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp -m GET -e remote_dropped -r 2 [ 1018.794631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp -d 192.168.202.50@tcp1 -m GET -e remote_dropped -r 2 [ 1018.811656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp -m GET -e remote_dropped -r 2 [ 1018.829862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.50@tcp1 -d 192.168.202.150@tcp1 -m GET -e remote_dropped -r 2 [ 1018.848228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp -m GET -e remote_dropped -r 2 [ 1018.864965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.50@tcp1 -d 192.168.202.50@tcp1 -m GET -e remote_dropped -r 2 [ 1018.886850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.50@tcp 192.168.202.150@tcp [ 1018.897760] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.150@tcp [ 1018.989632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.003112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.017778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.031366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.045794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.060103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.074137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.089794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.103431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.116982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.131676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.146050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.159841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.174308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.678316] LNet: 87819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.684467] LNetError: 87819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.687587] LNet: Removed LNI 192.168.202.50@tcp [ 1021.698057] LNet: Removed LNI 192.168.202.50@tcp1 [ 1021.920748] Key type .llcrypt unregistered [ 1021.923997] Key type ._llcrypt unregistered [ 1029.878355] Key type ._llcrypt registered [ 1029.881638] Key type .llcrypt registered [ 1030.597312] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:26:14 (1777044374) [ 1030.784035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.791840] alg: No test for adler32 (adler32-zlib) [ 1031.625848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.657782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1031.672359] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1032.103629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1032.121351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.50@tcp [ 1032.133332] LNetError: 88802:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.50@tcp to recovery [ 1032.186275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.50@tcp [ 1032.251610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.172118] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:26:19 (1777044379) [ 1035.214744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.227288] LNet: 89385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.230426] LNetError: 89385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.239651] LNet: Removed LNI 192.168.202.50@tcp [ 1035.258915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.297624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1035.309362] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1035.683483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1035.695805] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 1036.163438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.50@tcp [ 1036.176064] LNetError: 89656:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.50@tcp to recovery [ 1036.319404] LNet: 1 local NIs in recovery (showing 1): 192.168.202.50@tcp [ 1036.320930] LNet: There was an unexpected network error while writing to 192.168.202.50: rc = -22 [ 1037.279479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.50@tcp [ 1042.244471] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:26:26 (1777044386) [ 1043.038408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg250-server [ 1043.759200] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:26:27 (1777044387) [ 1044.581808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg250-server [ 1045.416825] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:26:29 (1777044389) [ 1046.304586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg250-server [ 1047.155357] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:26:31 (1777044391) [ 1047.243782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.256635] LNet: 91286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.260673] LNetError: 91286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.263812] LNet: Removed LNI 192.168.202.50@tcp [ 1047.272991] LNet: Removed LNI 192.168.202.50@tcp1 [ 1047.285588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.313445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1047.348581] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1047.714931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1047.744983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1047.768970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1047.808225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1047.830693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1047.846985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1050.948311] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:26:35 (1777044395) [ 1051.032050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.043386] LNet: 92012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.046421] LNetError: 92012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.050532] LNetError: 92012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1051.062807] LNet: Removed LNI 192.168.202.50@tcp [ 1051.072412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.096534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.107949] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1054.117806] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:26:38 (1777044398) [ 1054.173435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.188673] LNet: 92711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.193636] LNetError: 92711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.206924] LNet: Removed LNI 192.168.202.50@tcp [ 1054.219863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.243434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1054.257074] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1057.091469] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:26:41 (1777044401) [ 1057.752705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1058.438064] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:26:42 (1777044402) [ 1058.653223] LNet: 93748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.655683] LNetError: 93748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.663707] LNet: Removed LNI 192.168.202.50@tcp [ 1058.951123] Key type .llcrypt unregistered [ 1058.953174] Key type ._llcrypt unregistered [ 1061.435224] Key type ._llcrypt registered [ 1061.437958] Key type .llcrypt registered [ 1061.516263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.523074] alg: No test for adler32 (adler32-zlib) [ 1062.343732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.358756] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1065.712834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 1067.553067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.150@tcp [ 1067.703628] LNet: 94180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.707442] LNetError: 94180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.719977] LNet: Removed LNI 192.168.202.50@tcp [ 1067.954831] Key type .llcrypt unregistered [ 1067.956537] Key type ._llcrypt unregistered [ 1072.145152] Key type ._llcrypt registered [ 1072.146307] Key type .llcrypt registered [ 1072.687373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1073.315924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1074.008987] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:26:58 (1777044418) [ 1074.172476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.178299] alg: No test for adler32 (adler32-zlib) [ 1074.992821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.027940] LNet: 95265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.272724] Key type .llcrypt unregistered [ 1075.274181] Key type ._llcrypt unregistered [ 1077.674758] Key type ._llcrypt registered [ 1077.676072] Key type .llcrypt registered [ 1077.746498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.753169] alg: No test for adler32 (adler32-zlib) [ 1078.567388] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1078.569231] LNet: 95520:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1078.573659] LNet: 95520:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1078.586327] LNetError: 95532:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 1078.593431] LNetError: 95532:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.51@tcp has gone from up to down [ 1078.782356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1097.247168] LNetError: 95532:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.54@tcp has gone from up to down [ 1097.252125] LNetError: 95532:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 1097.255632] LNet: 95578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.258739] LNetError: 95578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.274684] LNet: Removed LNI 192.168.202.50@tcp [ 1097.285917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1097.310587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1097.321930] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1097.324554] LNet: 95588:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1097.328028] LNet: 95588:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1097.332146] LNetError: 95588:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.202.52@tcp is a gateway. Can not delete it [ 1097.339292] LNetError: 95588:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.202.52@tcp is a router. Can not be deleted [ 1097.344439] LNet: 95588:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.350421] LNetError: 95585:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.55@tcp is being used as a gateway but routing feature is not turned on [ 1097.356544] LNetError: 95585:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 1100.328845] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:27:24 (1777044444) [ 1100.384304] LNetError: 95589:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.202.54@tcp has gone from up to down [ 1100.394973] LNetError: 95589:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1101.127198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg250-server [ 1101.777726] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:27:26 (1777044446) [ 1102.440588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg250-server [ 1103.075306] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:27:27 (1777044447) [ 1103.762922] Lustre: Lustre: Build Version: 2.17.52_54_g344a3c0 [ 1105.455204] Key type lgssc registered [ 1105.989182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1114.835920] Key type lgssc unregistered [ 1114.991930] LNet: 98007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.995779] LNetError: 98007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.006615] LNet: Removed LNI 192.168.202.50@tcp [ 1115.243716] Key type .llcrypt unregistered [ 1115.244962] Key type ._llcrypt unregistered [ 1121.071161] Key type ._llcrypt registered [ 1121.072176] Key type .llcrypt registered [ 1121.625609] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:27:45 (1777044465) [ 1121.806722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.814742] alg: No test for adler32 (adler32-zlib) [ 1122.625702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.656129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1122.678231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1122.697785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1122.717578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1125.443168] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:27:49 (1777044469) [ 1125.479744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.491627] LNet: 99405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.509064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.547548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.559307] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1125.928301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.939158] LNetError: 99555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.947774] LNet: Removed LNI 192.168.202.50@tcp [ 1125.958358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.993389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1126.006457] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1126.213881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.225690] LNet: 99583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.227655] LNet: 99583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1126.239323] LNet: Removed LNI 192.168.202.50@tcp [ 1126.247610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.273427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1126.286299] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1129.002033] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:27:53 (1777044473) [ 1129.047791] LNet: 100167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.050378] LNetError: 100167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.055954] LNetError: 100167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1129.065978] LNet: Removed LNI 192.168.202.50@tcp [ 1129.339278] Key type .llcrypt unregistered [ 1129.341293] Key type ._llcrypt unregistered [ 1131.675371] Key type ._llcrypt registered [ 1131.676938] Key type .llcrypt registered [ 1131.775683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.783860] alg: No test for adler32 (adler32-zlib) [ 1132.618084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.661190] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1132.664218] LNet: *** cfs_fail_loc=e003, val=100*** [ 1133.049955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 1133.063395] Ping reply received with mapping: [ 1133.065128] found metadata reply NI[192.168.202.50@tcp] [ 1133.066781] 32 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1133.068508] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1133.070167] 5a 5a 5a 5a 5a 5a 5a 5a [ 1133.131690] LNet: 100580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.137154] LNetError: 100580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.149420] LNet: Removed LNI 192.168.202.50@tcp [ 1133.457770] Key type .llcrypt unregistered [ 1133.459679] Key type ._llcrypt unregistered [ 1138.069416] Key type ._llcrypt registered [ 1138.071038] Key type .llcrypt registered [ 1138.682257] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:28:02 (1777044482) [ 1138.902310] Key type .llcrypt unregistered [ 1138.904200] Key type ._llcrypt unregistered [ 1141.271224] Key type ._llcrypt registered [ 1141.272984] Key type .llcrypt registered [ 1141.381493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.389609] alg: No test for adler32 (adler32-zlib) [ 1142.310695] LNetError: 101623:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1142.495171] Key type .llcrypt unregistered [ 1142.497282] Key type ._llcrypt unregistered [ 1144.749415] Key type ._llcrypt registered [ 1144.750857] Key type .llcrypt registered [ 1145.336582] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:28:09 (1777044489) [ 1145.582457] Key type .llcrypt unregistered [ 1145.583881] Key type ._llcrypt unregistered [ 1148.266835] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1148.269620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1150.491301] Key type ._llcrypt registered [ 1150.492356] Key type .llcrypt registered [ 1150.566936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.574232] alg: No test for adler32 (adler32-zlib) [ 1151.418589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.460316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1156.754945] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:28:20 (1777044500) [ 1156.816973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1156.845054] LNet: 103397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.849470] LNetError: 103397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1156.868769] LNet: Removed LNI 10.1.2.3@tcp [ 1156.886285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1157.277417] Key type .llcrypt unregistered [ 1157.279585] Key type ._llcrypt unregistered [ 1160.061394] Key type ._llcrypt registered [ 1160.062984] Key type .llcrypt registered [ 1160.143394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.151254] alg: No test for adler32 (adler32-zlib) [ 1160.976853] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1161.008547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 1161.076428] LNet: 103691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.080462] LNetError: 103691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.091597] LNet: Removed LNI 192.168.202.50@tcp [ 1161.370561] Key type .llcrypt unregistered [ 1161.372080] Key type ._llcrypt unregistered [ 1165.858977] Key type ._llcrypt registered [ 1165.860690] Key type .llcrypt registered [ 1166.456167] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:28:30 (1777044510) [ 1166.681283] Key type .llcrypt unregistered [ 1166.682799] Key type ._llcrypt unregistered [ 1168.854340] Key type ._llcrypt registered [ 1168.855612] Key type .llcrypt registered [ 1168.950276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.958341] alg: No test for adler32 (adler32-zlib) [ 1172.458184] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:28:36 (1777044516) [ 1172.518355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1172.536169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.569957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.582751] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1172.919903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.935230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1172.950774] LNet: Removed LNI 192.168.202.50@tcp [ 1173.053451] LNet: 105431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.221547] Key type .llcrypt unregistered [ 1173.223204] Key type ._llcrypt unregistered [ 1177.928743] Key type ._llcrypt registered [ 1177.929908] Key type .llcrypt registered [ 1178.524853] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:28:42 (1777044522) [ 1178.728658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1178.736742] alg: No test for adler32 (adler32-zlib) [ 1179.552461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1179.573347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1179.589985] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1179.940253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 1179.957380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.50@tcp [ 1179.965235] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1179.966981] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1179.968175] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1179, 0, 0 [ 1179.970448] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1179.971688] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1179.973121] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1179.975070] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1179, 0, 0 [ 1179.978231] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1179.979707] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1179.981133] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1179.982982] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1179, 0, 1229 [ 1179.986026] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1179.987812] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1179.990144] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1179.991980] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1179, 0, 1229 [ 1179.995190] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1182.446052] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:28:46 (1777044526) [ 1182.629615] LNet: 106969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.638193] LNetError: 106969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.653661] LNet: Removed LNI 192.168.202.50@tcp [ 1182.918410] Key type .llcrypt unregistered [ 1182.923191] Key type ._llcrypt unregistered [ 1185.255994] Key type ._llcrypt registered [ 1185.258282] Key type .llcrypt registered [ 1185.346932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.352712] alg: No test for adler32 (adler32-zlib) [ 1186.164787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1186.177693] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1189.456670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 1192.501933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.518957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1192.529859] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 1192.888838] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1192.936033] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1192.939934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1192.942662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1195.077370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1195.090726] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1199.313389] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:29:03 (1777044543) [ 1199.345701] LNet: 108460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.350693] LNetError: 108460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.355207] LNet: Removed LNI 192.168.202.50@tcp [ 1200.415213] LNet: Removed LNI 192.168.202.50@tcp1 [ 1200.416879] LNetError: 108460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1200.419421] LNetError: 108460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1200.428601] LNet: Removed LNI 10.1.2.3@tcp99 [ 1200.670687] Key type .llcrypt unregistered [ 1200.672125] Key type ._llcrypt unregistered [ 1202.913490] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1202.915884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1205.106697] Key type ._llcrypt registered [ 1205.108075] Key type .llcrypt registered [ 1205.173726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.179483] alg: No test for adler32 (adler32-zlib) [ 1205.996911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.017431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1206.025189] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1206.283690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1206.292480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1207.041490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1207.068597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1207.089666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1207.100197] LNetError: 109212:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1207.115188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1207.134360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1207.291061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1207.318894] LNet: 109232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.321394] LNetError: 109232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.323768] LNet: Removed LNI 192.168.202.50@tcp [ 1207.331914] LNet: Removed LNI 10.1.2.3@tcp [ 1207.573490] Key type .llcrypt unregistered [ 1207.574874] Key type ._llcrypt unregistered [ 1211.288695] Key type ._llcrypt registered [ 1211.289780] Key type .llcrypt registered [ 1211.831557] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:29:16 (1777044556) [ 1211.963404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.969911] alg: No test for adler32 (adler32-zlib) [ 1212.777907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.803579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.815598] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1213.118812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.50@tcp [ 1213.149551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg250-client@tcp [ 1215.550986] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:29:19 (1777044559) [ 1215.599074] LNet: 110757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.601503] LNetError: 110757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.613431] LNet: Removed LNI 192.168.202.50@tcp [ 1215.868588] Key type .llcrypt unregistered [ 1215.869822] Key type ._llcrypt unregistered [ 1217.912430] Key type ._llcrypt registered [ 1217.913782] Key type .llcrypt registered [ 1217.976545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.983147] alg: No test for adler32 (adler32-zlib) [ 1218.785814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1218.797395] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1218.810933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1221.108157] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:29:25 (1777044565) [ 1221.626457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1222.204571] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:29:26 (1777044566) [ 1222.731599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1223.300721] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:29:27 (1777044567) [ 1223.331801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1223.343217] LNet: 112174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.346814] LNetError: 112174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.356350] LNet: Removed LNI 192.168.202.50@tcp [ 1223.365716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1223.393231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1223.404529] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1223.703744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1223.725034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1226.331480] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:29:30 (1777044570) [ 1226.363528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.377443] LNet: 112882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.381962] LNetError: 112882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.392985] LNet: Removed LNI 192.168.202.50@tcp [ 1226.414152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.439343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1226.455302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1226.711582] Key type .llcrypt unregistered [ 1226.713182] Key type ._llcrypt unregistered [ 1230.664280] Key type ._llcrypt registered [ 1230.665605] Key type .llcrypt registered [ 1231.167650] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:29:35 (1777044575) [ 1231.308990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.316183] alg: No test for adler32 (adler32-zlib) [ 1232.137164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.162412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1232.172379] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1232.479915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1232.503689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1234.929507] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:29:39 (1777044579) [ 1234.969297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1234.979341] LNet: 114437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.982238] LNetError: 114437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.994306] LNet: Removed LNI 192.168.202.50@tcp [ 1235.005544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.030605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1235.043511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1237.291730] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:29:41 (1777044581) [ 1237.327690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.335889] LNet: 115017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.344861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.375573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1237.512452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1237.527484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1237.551757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1237.693494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1237.706672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1237.739130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.754511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.780542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1237.934667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1237.949336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1237.977500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1238.115867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1238.126172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1240.394625] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:29:44 (1777044584) [ 1240.442774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.452949] LNet: 115652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.455629] LNet: 115652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1240.469888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.492807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1240.503339] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1240.759631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1240.767438] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1241.077873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1241.101140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1241.182414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.190500] LNetError: 115933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.192800] LNet: Removed LNI 192.168.202.50@tcp [ 1241.198299] LNet: Removed LNI 192.168.202.50@tcp2 [ 1241.204867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.227104] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1241.230536] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1241.233296] LNet: 115939:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.400494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1241.477454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.527242] LNet: Removed LNI 192.168.202.50@tcp [ 1242.528895] LNetError: 115976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.531607] LNetError: 115976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1242.541436] LNet: Removed LNI 192.168.202.50@tcp2 [ 1242.552453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.578878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.591554] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1242.885578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1242.895776] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1243.256139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1243.280047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1243.307245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1243.391964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.403528] LNet: 116260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.407094] LNet: 116260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1244.447139] LNet: Removed LNI 192.168.202.50@tcp [ 1245.481469] LNet: Removed LNI 192.168.202.50@tcp2 [ 1245.490980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.518052] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1245.521686] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1245.525565] LNet: 116267:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.713385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1245.790881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.801520] LNetError: 116304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.804191] LNetError: 116304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1245.807207] LNet: Removed LNI 192.168.202.50@tcp [ 1246.821287] LNet: Removed LNI 192.168.202.50@tcp2 [ 1246.832536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.857661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1246.870612] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1247.173824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1247.184797] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1247.580603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1247.611831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1247.703346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.716244] LNet: 116583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.718357] LNet: 116583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1248.735205] LNet: Removed LNI 192.168.202.50@tcp [ 1248.744166] LNet: Removed LNI 192.168.202.50@tcp2 [ 1248.753962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.779343] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1248.782443] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1248.785552] LNet: 116589:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.986160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1249.073431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.143137] LNet: Removed LNI 192.168.202.50@tcp [ 1250.144460] LNetError: 116626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.146780] LNetError: 116626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1250.157655] LNet: Removed LNI 192.168.202.50@tcp2 [ 1250.166710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.191402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1250.202047] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1250.473738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1250.483964] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1250.842829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1250.859919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1250.884791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1250.965543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.999162] LNet: Removed LNI 192.168.202.50@tcp [ 1253.029308] LNet: Removed LNI 192.168.202.50@tcp2 [ 1253.039604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.076175] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1253.080069] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1253.083655] LNet: 116917:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.283543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1253.370827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.380991] LNet: Removed LNI 192.168.202.50@tcp [ 1254.440423] LNet: Removed LNI 192.168.202.50@tcp2 [ 1254.450047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.482503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1254.495573] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1254.833152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1254.842459] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1255.214300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1255.235114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1255.269842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1255.438451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1256.479140] LNet: Removed LNI 192.168.202.50@tcp [ 1256.487639] LNet: Removed LNI 192.168.202.50@tcp2 [ 1256.500384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.528084] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1256.531536] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1256.728146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1256.887907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1256.899875] LNet: 117308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.902896] LNet: 117308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1257.951206] LNet: Removed LNI 192.168.202.50@tcp [ 1257.959632] LNet: Removed LNI 192.168.202.50@tcp2 [ 1257.969948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.000725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1258.016659] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1258.335306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1258.344592] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1258.757548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1258.780419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1258.795992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1258.823254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1258.979930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1258.989128] LNetError: 117610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.991534] LNetError: 117610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1258.994823] LNet: Removed LNI 192.168.202.50@tcp [ 1260.013615] LNet: Removed LNI 192.168.202.50@tcp2 [ 1260.025236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.063538] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1260.068490] LNet: Added LNI 192.168.202.50@tcp2 [8/256/0/180] [ 1260.073565] LNet: 117616:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.079263] LNet: 117616:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1260.296279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1263.410897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1264.095060] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:30:08 (1777044608) [ 1264.155507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.168283] LNet: Removed LNI 192.168.202.50@tcp [ 1265.189497] LNet: Removed LNI 192.168.202.50@tcp2 [ 1265.198435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.310806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.323395] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1265.639265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1265.649232] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1266.015824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1266.026856] LNet: Removed LNI 192.168.202.50@tcp [ 1266.034351] LNet: Removed LNI 10.1.2.3@tcp [ 1266.041888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.067938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1266.079569] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1266.084908] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1266.252294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1266.261401] LNet: Removed LNI 192.168.202.50@tcp [ 1266.271405] LNet: Removed LNI 10.1.2.3@tcp [ 1268.730601] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:30:13 (1777044613) [ 1268.779948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.796333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1268.838586] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1268.889750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1268.891793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1268.894590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1271.064269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.080896] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1271.386544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1271.395215] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1272.224127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.50@tcp [ 1272.244385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1272.271612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.50@tcp -m PUT -e local_timeout -r 1 [ 1272.297802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1272.319482] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1272.321195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1272.428504] LNetError: 119312:0:(lib-move.c:4969:lnet_parse()) 192.168.202.50@tcp, src 192.168.202.50@tcp: Dropping PUT (error -108 looking up sender) [ 1273.439366] LNet: 119772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.442589] LNet: 119772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1274.463107] LNet: Removed LNI 192.168.202.50@tcp [ 1274.470464] LNet: Removed LNI 10.1.2.3@tcp [ 1274.660179] Key type .llcrypt unregistered [ 1274.661667] Key type ._llcrypt unregistered [ 1279.207837] Key type ._llcrypt registered [ 1279.208913] Key type .llcrypt registered [ 1279.769777] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:30:23 (1777044623) [ 1279.933293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.940875] alg: No test for adler32 (adler32-zlib) [ 1280.765468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.937535] LNet: 120623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.187387] Key type .llcrypt unregistered [ 1281.188636] Key type ._llcrypt unregistered [ 1283.259826] Key type ._llcrypt registered [ 1283.261706] Key type .llcrypt registered [ 1283.339067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.344843] alg: No test for adler32 (adler32-zlib) [ 1284.158641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1284.172252] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1287.053523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 1287.114313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1287.321831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.150@tcp [ 1287.410608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.424382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1287.440860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.460699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.479545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.492089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.505943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.520457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.530477] LNet: 121071:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.539840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.557277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.573335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.587960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.602798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.618530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.844952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1287.925291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.939552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1287.955901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.969938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.985687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.001854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.013779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.024760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.032919] LNet: 121189:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.042133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.054893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.068820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.081948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.095848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.110358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.333257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.150@tcp [ 1288.414124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.426546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1288.439536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.452764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.465306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.480664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.493876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.507553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.520636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.533728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.547936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.561096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.574388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.591196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.825585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1288.921511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.935592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1288.949563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.966776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.982464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.997040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.011637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.024605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.035471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.046832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.061809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.075742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.091896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.107441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.216406] LNet: 121469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.218904] LNetError: 121469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.228630] LNet: Removed LNI 192.168.202.50@tcp [ 1289.358552] Key type .llcrypt unregistered [ 1289.359517] Key type ._llcrypt unregistered [ 1294.857918] Key type ._llcrypt registered [ 1294.858952] Key type .llcrypt registered [ 1295.305937] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:30:39 (1777044639) [ 1295.426327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.431930] alg: No test for adler32 (adler32-zlib) [ 1296.240431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.386342] LNet: 122354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.611507] Key type .llcrypt unregistered [ 1296.612796] Key type ._llcrypt unregistered [ 1298.551993] Key type ._llcrypt registered [ 1298.552873] Key type .llcrypt registered [ 1298.617363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1298.623212] alg: No test for adler32 (adler32-zlib) [ 1299.419615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1299.428518] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1302.198578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.150@tcp [ 1304.727109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.740106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.748673] LNet: Added LNI 192.168.202.50@tcp1 [8/256/0/180] [ 1305.086560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1305.280789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.150@tcp [ 1305.373162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.385777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.400633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.414735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.430534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.446149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.460460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.473805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.482347] LNet: 123004:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.491161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.507375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.523559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.537855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.554140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.570342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.812456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1305.903215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.918541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.934353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.947575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.962478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.978730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.995200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.010850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.019507] LNet: 123121:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.029608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.046632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.061445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.075602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.092654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.108783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.359653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.150@tcp [ 1306.455791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.467246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.481731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.495609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.512179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.524245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.536991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.551173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.566170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.580581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.592275] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.150@tcp [ 1306.595943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.618127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.638320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.655780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.879973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1306.960067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.973584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.988733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.001422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.012161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.026634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.037908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.049224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.055821] LNet: 123355:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.059373] LNet: 123355:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1307.067589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.081633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.092980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.105930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.120705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.133534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.221796] LNet: 123436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.225237] LNetError: 123436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.228913] LNet: Removed LNI 192.168.202.50@tcp [ 1309.239544] LNet: Removed LNI 192.168.202.50@tcp1 [ 1309.387738] Key type .llcrypt unregistered [ 1309.391794] Key type ._llcrypt unregistered [ 1314.316849] Key type ._llcrypt registered [ 1314.318063] Key type .llcrypt registered [ 1314.758335] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:30:59 (1777044659) [ 1317.143649] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:31:01 (1777044661) [ 1319.482359] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:31:03 (1777044663) [ 1322.059139] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:31:06 (1777044666) [ 1324.418159] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:31:08 (1777044668) [ 1326.727012] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:31:11 (1777044671) [ 1329.056445] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:31:13 (1777044673) [ 1331.659775] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:31:15 (1777044675) [ 1334.138378] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:31:18 (1777044678) [ 1336.376246] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:31:20 (1777044680) [ 1338.741681] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:31:23 (1777044683) [ 1341.147927] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:31:25 (1777044685) [ 1343.327911] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:31:27 (1777044687) [ 1345.578750] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:31:29 (1777044689) [ 1347.988416] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:31:32 (1777044692) [ 1350.415983] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:31:34 (1777044694) [ 1352.736090] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:31:37 (1777044697) [ 1355.085775] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:31:39 (1777044699) [ 1355.223811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.230869] alg: No test for adler32 (adler32-zlib) [ 1356.038872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.089345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.100588] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1356.517582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.525963] LNet: 134131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.528535] LNetError: 134131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.536402] LNet: Removed LNI 192.168.202.50@tcp [ 1356.545796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.571722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1356.583829] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1359.088719] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:31:43 (1777044703) [ 1361.994024] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:31:46 (1777044706) [ 1364.724570] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:31:49 (1777044709) [ 1367.134837] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:31:51 (1777044711) [ 1369.409182] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:31:53 (1777044713) [ 1371.801489] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:31:56 (1777044716) [ 1374.172246] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:31:58 (1777044718) [ 1376.633382] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:32:00 (1777044720) [ 1379.062472] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:32:03 (1777044723) [ 1381.481108] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:32:05 (1777044725) [ 1383.866840] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:32:08 (1777044728) [ 1386.412406] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:32:10 (1777044730) [ 1389.093691] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1186 sec ========== 11:32:13 (1777044733) [ 1389.210753] LNet: 141588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.213722] LNetError: 141588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.220618] LNet: Removed LNI 192.168.202.50@tcp [ 1389.455615] Key type .llcrypt unregistered [ 1389.456953] Key type ._llcrypt unregistered [ 1394.067529] Key type ._llcrypt registered [ 1394.069060] Key type .llcrypt registered [ 1394.264861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.272043] alg: No test for adler32 (adler32-zlib) [ 1395.142408] Lustre: Lustre: Build Version: 2.17.52_54_g344a3c0 [ 1395.241940] LNet: Added LNI 192.168.202.50@tcp [8/256/0/180] [ 1396.831211] Key type lgssc registered [ 1397.250825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.672766] Lustre: Mounted lustre-client [ 1422.175512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1429.492753] LustreError: 144025:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8e3a893dc800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1429.517541] Lustre: Unmounted lustre-client [ 1441.398515] Key type lgssc unregistered [ 1441.515582] LNet: 144505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.518920] LNetError: 144505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.529796] LNet: Removed LNI 192.168.202.50@tcp [ 1441.796142] Key type .llcrypt unregistered [ 1441.797600] Key type ._llcrypt unregistered