[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 380912249 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.007224] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008013] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.012033] Yama: becoming mindful. [ 0.012681] SELinux: Initializing. [ 0.013050] *** VALIDATE selinux *** [ 0.021073] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025459] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026100] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027078] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028079] *** VALIDATE tmpfs *** [ 0.029434] *** VALIDATE proc *** [ 0.030186] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032208] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034057] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.040592] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.042122] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043494] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044010] ... version: 2 [ 0.045006] ... bit width: 48 [ 0.046005] ... generic registers: 4 [ 0.047005] ... value mask: 0000ffffffffffff [ 0.048006] ... max period: 00007fffffffffff [ 0.049005] ... fixed-purpose events: 3 [ 0.050005] ... event mask: 000000070000000f [ 0.051937] rcu: Hierarchical SRCU implementation. [ 0.053282] smp: Bringing up secondary CPUs ... [ 0.054428] x86: Booting SMP configuration: [ 0.055011] .... node #0, CPUs: #1 #2 #3 [ 0.057736] smp: Brought up 1 node, 4 CPUs [ 0.059006] smpboot: Max logical packages: 1 [ 0.060008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.275137] node 0 deferred pages initialised in 214ms [ 0.279227] devtmpfs: initialized [ 0.280288] x86/mm: Memory block size: 128MB [ 0.283423] gcov: version magic: 0x41383552 [ 0.285332] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287354] pinctrl core: initialized pinctrl subsystem [ 0.288213] [ 0.288802] ************************************************************* [ 0.289011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290007] ** ** [ 0.291009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292009] ** ** [ 0.293007] ** This means that this kernel is built to expose internal ** [ 0.294007] ** IOMMU data structures, which may compromise security on ** [ 0.295006] ** your system. ** [ 0.296007] ** ** [ 0.297008] ** If you see this message and you are not debugging the ** [ 0.298006] ** kernel, report this immediately to your vendor! ** [ 0.299006] ** ** [ 0.300005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301006] ************************************************************* [ 0.302568] NET: Registered protocol family 16 [ 0.303410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.306537] cpuidle: using governor menu [ 0.307772] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.310465] PCI: Using configuration type 1 for base access [ 0.311121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.322017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325071] cryptd: max_cpu_qlen set to 1000 [ 0.326200] ACPI: Added _OSI(Module Device) [ 0.328009] ACPI: Added _OSI(Processor Device) [ 0.328939] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.332773] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.336364] ACPI: Interpreter enabled [ 0.337047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338010] ACPI: Using IOAPIC for interrupt routing [ 0.340134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352242] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.355009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360185] acpiphp: Slot [2] registered [ 0.362095] acpiphp: Slot [3] registered [ 0.363068] acpiphp: Slot [4] registered [ 0.364088] acpiphp: Slot [5] registered [ 0.365075] acpiphp: Slot [6] registered [ 0.367079] acpiphp: Slot [7] registered [ 0.368085] acpiphp: Slot [8] registered [ 0.369070] acpiphp: Slot [9] registered [ 0.370060] acpiphp: Slot [10] registered [ 0.371057] acpiphp: Slot [11] registered [ 0.372065] acpiphp: Slot [12] registered [ 0.374071] acpiphp: Slot [13] registered [ 0.375064] acpiphp: Slot [14] registered [ 0.376093] acpiphp: Slot [15] registered [ 0.377065] acpiphp: Slot [16] registered [ 0.378070] acpiphp: Slot [17] registered [ 0.380089] acpiphp: Slot [18] registered [ 0.381058] acpiphp: Slot [19] registered [ 0.382063] acpiphp: Slot [20] registered [ 0.383081] acpiphp: Slot [21] registered [ 0.385063] acpiphp: Slot [22] registered [ 0.386045] acpiphp: Slot [23] registered [ 0.387040] acpiphp: Slot [24] registered [ 0.387798] acpiphp: Slot [25] registered [ 0.388119] acpiphp: Slot [26] registered [ 0.390063] acpiphp: Slot [27] registered [ 0.391064] acpiphp: Slot [28] registered [ 0.392077] acpiphp: Slot [29] registered [ 0.392905] acpiphp: Slot [30] registered [ 0.393050] acpiphp: Slot [31] registered [ 0.393893] PCI host bridge to bus 0000:00 [ 0.395013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.397025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.398011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.401013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.403015] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.405024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.407147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410400] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.414229] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420494] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.424056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.426010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.428011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.430013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.432550] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434716] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.437046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439523] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.452939] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.457016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.459000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.476227] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487961] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.501752] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.514026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.522022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.534018] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.541587] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.544321] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.545296] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.548353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.550216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.554129] iommu: Default domain type: Passthrough [ 0.556473] SCSI subsystem initialized [ 0.558094] ACPI: bus type USB registered [ 0.559093] usbcore: registered new interface driver usbfs [ 0.561049] usbcore: registered new interface driver hub [ 0.563075] usbcore: registered new device driver usb [ 0.564154] pps_core: LinuxPPS API ver. 1 registered [ 0.566025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.569109] PTP clock support registered [ 0.573178] EDAC MC: Ver: 3.0.0 [ 0.575194] PCI: Using ACPI for IRQ routing [ 0.577662] NetLabel: Initializing [ 0.578006] NetLabel: domain hash size = 128 [ 0.578939] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.580071] NetLabel: unlabeled traffic allowed by default [ 0.582142] vgaarb: loaded [ 0.583260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.585010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.589278] clocksource: Switched to clocksource kvm-clock [ 0.692215] VFS: Disk quotas dquot_6.6.0 [ 0.693406] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.695095] *** VALIDATE ramfs *** [ 0.696452] *** VALIDATE hugetlbfs *** [ 0.698625] pnp: PnP ACPI init [ 0.700129] pnp: PnP ACPI: found 6 devices [ 0.714385] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.717397] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.719172] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.720258] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.721642] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.723571] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.726046] NET: Registered protocol family 2 [ 0.727547] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.732345] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.734864] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.738739] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.741173] TCP: Hash tables configured (established 65536 bind 65536) [ 0.743287] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.745345] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.749449] NET: Registered protocol family 1 [ 0.751273] RPC: Registered named UNIX socket transport module. [ 0.752693] RPC: Registered udp transport module. [ 0.753780] RPC: Registered tcp transport module. [ 0.754884] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.756416] NET: Registered protocol family 44 [ 0.757496] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.758928] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.760344] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.761811] PCI: CLS 0 bytes, default 64 [ 0.762983] Unpacking initramfs... [ 2.193547] debug: unmapping init [mem 0xffff8bafbcc64000-0xffff8bafbffcffff] [ 2.197885] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.200265] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.203074] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.823687] Initialise system trusted keyrings [ 2.825162] Key type blacklist registered [ 2.828762] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.838070] zbud: loaded [ 2.846661] *** VALIDATE nfs *** [ 2.847672] *** VALIDATE nfs4 *** [ 2.848840] pstore: using deflate compression [ 2.852191] Platform Keyring initialized [ 2.999631] NET: Registered protocol family 38 [ 3.003880] Key type asymmetric registered [ 3.007162] Asymmetric key parser 'x509' registered [ 3.008704] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.011956] io scheduler mq-deadline registered [ 3.013552] io scheduler kyber registered [ 3.014710] io scheduler bfq registered [ 3.016957] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.020460] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.022959] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.025459] ACPI: Power Button [PWRF] [ 3.157738] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.296968] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.503023] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.539247] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.576268] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.582438] Non-volatile memory driver v1.3 [ 3.584121] Linux agpgart interface v0.103 [ 3.629401] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.632126] vda: detected capacity change from 0 to 68571136 [ 3.671298] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.673537] vdb: detected capacity change from 0 to 1073741824 [ 3.686331] libphy: Fixed MDIO Bus: probed [ 3.706187] usbcore: registered new interface driver usbserial_generic [ 3.708504] usbserial: USB Serial support registered for generic [ 3.712169] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.724392] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.726450] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.729835] mousedev: PS/2 mouse device common for all mice [ 3.733505] rtc_cmos 00:05: RTC can wake from S4 [ 3.736600] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.744058] rtc_cmos 00:05: registered as rtc0 [ 3.744139] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.744174] intel_pstate: CPU model not supported [ 3.747150] hid: raw HID events driver (C) Jiri Kosina [ 3.759134] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.759773] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.772812] usbcore: registered new interface driver usbhid [ 3.774362] usbhid: USB HID core driver [ 3.775580] drop_monitor: Initializing network drop monitor service [ 3.782838] Initializing XFRM netlink socket [ 3.784703] NET: Registered protocol family 10 [ 3.789879] Segment Routing with IPv6 [ 3.791288] NET: Registered protocol family 17 [ 3.798166] mpls_gso: MPLS GSO support [ 3.806925] RAS: Correctable Errors collector initialized. [ 3.812249] AVX version of gcm_enc/dec engaged. [ 3.813893] AES CTR mode by8 optimization enabled [ 4.007257] sched_clock: Marking stable (4007237801, 0)->(4879235535, -871997734) [ 4.010921] registered taskstats version 1 [ 4.013323] Loading compiled-in X.509 certificates [ 4.015604] zswap: loaded using pool lzo/zbud [ 4.055633] Key type big_key registered [ 4.073648] Key type encrypted registered [ 4.075559] ima: No TPM chip found, activating TPM-bypass! [ 4.079049] ima: Allocated hash algorithm: sha1 [ 4.080898] ima: No architecture policies found [ 4.086262] evm: Initialising EVM extended attributes: [ 4.088414] evm: security.selinux [ 4.089920] evm: security.ima [ 4.091406] evm: security.capability [ 4.092911] evm: HMAC attrs: 0x1 [ 4.095621] rtc_cmos 00:05: setting system clock to 2025-12-03 03:39:57 UTC (1764733197) [ 4.104422] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 4.107866] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 4.117426] Write protecting the kernel read-only data: 28672k [ 4.131705] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 4.137712] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 4.182238] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.191498] systemd[1]: Detected virtualization kvm. [ 4.193618] systemd[1]: Detected architecture x86-64. [ 4.195651] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.240410] systemd[1]: No hostname configured. [ 4.242039] systemd[1]: Set hostname to . [ 4.243725] random: systemd: uninitialized urandom read (16 bytes read) [ 4.246063] systemd[1]: Initializing machine ID from random generator. [ 4.337620] random: ln: uninitialized urandom read (6 bytes read) [ 4.497793] random: systemd: uninitialized urandom read (16 bytes read) [ 4.500690] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.504717] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.509615] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.788968] device-mapper: uevent: version 1.0.3 [ 5.793538] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.419832] virtio_net virtio0 ens2: renamed from eth0 [ 7.740980] scsi host0: ata_piix [ 7.783725] scsi host1: ata_piix [ 7.785358] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.792487] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.868220] random: fast init done [ 12.607700] random: crng init done [ 12.613127] random: 7 urandom warning(s) missed due to ratelimiting [ 14.968189] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.044425] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.064679] printk: systemd: 25 output lines suppressed due to ratelimiting [ 18.525951] SELinux: Disabled at runtime. [ 18.605594] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.619180] systemd[1]: Detected virtualization kvm. [ 18.620536] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.625931] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.630385] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.637283] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.641082] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.644720] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.664868] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.673102] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices...[ 19.993430] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [[ 20.845571] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.921809] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.943818] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.467948] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.691155] EDAC sbridge: Ver: 1.1.2 [ 26.624788] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 27.083914] NFS: Registering the id_resolver key type [ 27.086320] Key type id_resolver registered [ 27.088542] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-client login: [ 66.435193] hrtimer: interrupt took 5184973 ns [ 77.389652] libcfs: loading out-of-tree module taints kernel. [ 77.798620] Key type ._llcrypt registered [ 77.806876] Key type .llcrypt registered [ 78.439414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 78.470650] alg: No test for adler32 (adler32-zlib) [ 79.978934] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 81.176420] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 82.928229] Key type lgssc registered [ 84.571465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 188.617858] Lustre: Mounted lustre-client [ 192.890238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.078420] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing check_logdir /tmp/testlogs/ [ 208.644000] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing yml_node [ 211.818854] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 213.976789] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 214.496295] Lustre: lustre-OST0000-osc-ffff8bb007083000: disconnect after 23s idle [ 216.161936] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 217.421636] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Dec 2 22:43:29 EST 2025 [ 240.611976] LustreError: 7911:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8bb007083000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 240.647543] LustreError: 7911:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 240.704107] Lustre: Unmounted lustre-client [ 272.907357] Key type lgssc unregistered [ 273.310269] LNet: 8359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.317753] LNetError: 8359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 273.342888] LNet: Removed LNI 192.168.204.9@tcp [ 274.274838] Key type .llcrypt unregistered [ 274.278945] Key type ._llcrypt unregistered [ 303.220665] Key type ._llcrypt registered [ 303.222242] Key type .llcrypt registered [ 303.417363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 303.438903] alg: No test for adler32 (adler32-zlib) [ 304.379261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 304.426587] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 304.441199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 308.235134] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 308.452557] LNet: 9234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.461392] LNetError: 9234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.485474] LNet: Removed LNI 192.168.204.9@tcp [ 309.042329] Key type .llcrypt unregistered [ 309.044625] Key type ._llcrypt unregistered [ 314.323924] Key type ._llcrypt registered [ 314.325439] Key type .llcrypt registered [ 315.762599] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:45:07 (1764733507) [ 316.250280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 316.269825] alg: No test for adler32 (adler32-zlib) [ 317.272853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.353373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.387085] LNet: 9640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.409620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.474675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.495812] LNet: 9647:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 324.228728] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:45:16 (1764733516) [ 324.490291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.506831] LNet: 10225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.554757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.650203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 331.151646] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:45:23 (1764733523) [ 331.214619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.260353] LNet: 10802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.290695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.383632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 337.621576] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:45:29 (1764733529) [ 337.887197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.916474] LNet: 11397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.934902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.995947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 344.748767] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:45:36 (1764733536) [ 344.991302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.010596] LNet: 11983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.026864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.082390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 345.378335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 351.537927] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:45:43 (1764733543) [ 351.804685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.857400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.949949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.453455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.785666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 359.456294] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:45:51 (1764733551) [ 359.564903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.609634] LNet: 13163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.612942] LNet: 13163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 359.657346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.766097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 366.122541] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:45:58 (1764733558) [ 366.241794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.285594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.402958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 366.461901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 366.918618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 366.977666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 367.547989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 367.613572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 368.007099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 368.056726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 368.492069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 368.538841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.108272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 369.158940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 369.652444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 369.719210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 370.156739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 370.197871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 370.552380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 370.605593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 371.175756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 371.222638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 371.756073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 371.814525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 372.369795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 372.429845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 372.941434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 372.978571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.542673] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:46:11 (1764733571) [ 379.634936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.670830] LNet: 14429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.679144] LNet: 14429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 379.700735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.772262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 379.817276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 387.208690] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:46:19 (1764733579) [ 387.319117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.389451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.465367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 387.515907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.084715] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:46:26 (1764733586) [ 394.301227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.359755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.451535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 394.477738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 401.078846] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:46:32 (1764733592) [ 401.197369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.258298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.348328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 401.393072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 408.170884] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:46:40 (1764733600) [ 408.237348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.295764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.444679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 408.511127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 415.326927] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:46:46 (1764733606) [ 415.443548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.478985] LNet: 17359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.494287] LNet: 17359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 415.546226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.686685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 415.728172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 422.845181] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:46:54 (1764733614) [ 422.974109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.030628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.165117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.611974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.402712] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:47:03 (1764733623) [ 431.477213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.527036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.890294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.292978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.273513] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:47:11 (1764733631) [ 439.562130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.616391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.739789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.945644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.816779] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:47:18 (1764733638) [ 446.948989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.009269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.520848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.840215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.846255] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:47:26 (1764733646) [ 455.001161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.094255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.267863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.315664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.038827] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:47:34 (1764733654) [ 462.135115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.190600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.273734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.320503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.552040] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:47:41 (1764733661) [ 469.938783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.008642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.148944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 476.973853] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:47:48 (1764733668) [ 477.163466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.203236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.340593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 485.350726] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:47:57 (1764733677) [ 485.470615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.503354] LNet: 22795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.507785] LNet: 22795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 485.544912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.670689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 486.218699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 494.238877] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:48:05 (1764733685) [ 494.352509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.395962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.529256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 494.599483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 501.391961] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:48:13 (1764733693) [ 501.580302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.646678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.761804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 501.788917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 508.229822] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:48:20 (1764733700) [ 508.523573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.576337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.707355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 508.751907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 516.213607] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:48:28 (1764733708) [ 516.494919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.555163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.632974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.662706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 523.526447] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:48:35 (1764733715) [ 523.620324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.713563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.858799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 523.956943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 523.973653] LNetError: 25751:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 524.013589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 524.050610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 530.517914] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:48:42 (1764733722) [ 530.636743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.695298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.857036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cmbuf1XpSU [ 530.981722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cmbuf1XpSU [ 531.052521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.cmbuf1XpSU [ 537.914501] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:48:49 (1764733729) [ 538.213926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.268106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.373492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 538.446629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 538.558886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 545.511805] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:48:57 (1764733737) [ 545.636331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.711358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.798737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 549.532881] LNet: Message forwarding will be enabled in 2s [ 552.610573] LNet: Message forwarding enabled [ 553.141620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.177334] LNet: Message forwarding will stop in 2s to 6s [ 557.785330] LNet: Message forwarding disabled [ 557.804644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.544330] LNet: Message forwarding will be enabled in 2s [ 563.936826] LNet: Message forwarding enabled [ 568.058821] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:49:19 (1764733759) [ 568.352868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.471840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.525790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 576.722900] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:49:28 (1764733768) [ 576.836520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.867852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.927524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 576.970140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 576.997355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 577.065950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 577.125785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 577.163347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 577.223552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.286942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 577.356158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 577.411378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 577.485761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 577.530245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 577.601700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 577.667729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 577.723413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 577.760666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 577.804476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 577.852730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.922310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 577.958585] LNetError: 28822:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 584.124829] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:49:36 (1764733776) [ 584.242070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.346974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.474676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.226760] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:49:43 (1764733783) [ 594.232434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg409-server [ 596.288353] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:49:48 (1764733788) [ 596.545794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.584385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.626226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.649688] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 598.297455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[10-14]@tcp [ 598.333089] LNet: 30539:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 598.433979] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.11@tcp is being used as a gateway but routing feature is not turned on [ 598.464211] LNetError: 30329:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.11@tcp has gone from up to down [ 599.446846] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 599.452376] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 600.484537] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 600.499235] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 602.531897] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 602.536371] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 604.986024] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:49:56 (1764733796) [ 606.640267] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 606.645158] LNetError: 30329:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 606.850479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg409-server [ 609.247381] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:50:00 (1764733800) [ 609.367280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.380261] LNetError: 31466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.470501] LNet: Removed LNI 192.168.204.9@tcp [ 609.567146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.719181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.766139] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 610.588564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 610.622202] LNet: 31591:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 610.629545] LNet: 31591:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 610.650964] LNetError: 31478:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.12@tcp has gone from up to down [ 610.661735] LNetError: 31478:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 610.671517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 618.062540] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:50:09 (1764733809) [ 618.188801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.280361] LNet: 32176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.282779] LNet: 32176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 618.304321] LNetError: 32176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.325879] LNet: Removed LNI 192.168.204.9@tcp [ 618.379558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.593461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 618.724655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.764689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.848097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.929257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.997358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.088827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 619.202937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.253590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.321658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.398192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.450488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.519448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 619.668983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.700312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.788436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.881814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.943565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.020099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 620.160643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.225962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.288081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.353250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.384949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.444860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 620.556651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.626584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.704471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.831940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.883522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.953659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 621.041961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.100229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.192742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 627.952959] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:50:19 (1764733819) [ 629.577778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg409-server [ 631.642377] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:50:23 (1764733823) [ 633.592397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg409-server [ 635.544706] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:50:27 (1764733827) [ 636.744583] Key type .llcrypt unregistered [ 636.759068] Key type ._llcrypt unregistered [ 642.599564] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 642.804727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 642.807443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 642.810050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 645.641544] Key type ._llcrypt registered [ 645.644714] Key type .llcrypt registered [ 645.922290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.940258] alg: No test for adler32 (adler32-zlib) [ 646.920765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.002622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.031220] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 647.778721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 647.817900] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.734250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 650.758188] LNet: Removed LNI 10.1.2.3@tcp [ 650.881192] LNet: 34302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.891489] LNetError: 34302:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.912432] LNet: Removed LNI 192.168.204.9@tcp [ 651.694520] Key type .llcrypt unregistered [ 651.699068] Key type ._llcrypt unregistered [ 663.503142] Key type ._llcrypt registered [ 663.506251] Key type .llcrypt registered [ 665.139959] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:50:56 (1764733856) [ 665.644198] Key type .llcrypt unregistered [ 665.646070] Key type ._llcrypt unregistered [ 671.582385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 671.818995] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 671.826143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 671.832683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 674.586394] Key type ._llcrypt registered [ 674.588430] Key type .llcrypt registered [ 674.763738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 674.785984] alg: No test for adler32 (adler32-zlib) [ 675.763861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.851579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.909037] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 676.665171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.679160] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.575627] LNet: 35791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.580215] LNetError: 35791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.592600] LNet: Removed LNI 192.168.204.9@tcp [ 679.605904] LNet: Removed LNI 10.1.2.3@tcp [ 680.454900] Key type .llcrypt unregistered [ 680.456705] Key type ._llcrypt unregistered [ 692.433509] Key type ._llcrypt registered [ 692.435160] Key type .llcrypt registered [ 693.777634] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:51:25 (1764733885) [ 694.240481] Key type .llcrypt unregistered [ 694.242264] Key type ._llcrypt unregistered [ 700.050016] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.253766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.257794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.264921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 702.974326] Key type ._llcrypt registered [ 702.993302] Key type .llcrypt registered [ 703.249305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.267057] alg: No test for adler32 (adler32-zlib) [ 704.245921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.329436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 704.382467] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 706.490373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 706.501075] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.836945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 708.874880] LNet: Removed LNI 10.1.2.3@tcp [ 708.882893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 708.977726] LNet: Removed LNI 10.1.2.31@tcp [ 709.011232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 709.037323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.700984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 709.718126] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 710.741983] LNet: 37702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.750596] LNetError: 37702:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 710.779228] LNet: Removed LNI 10.1.2.3@tcp [ 710.804405] LNet: Removed LNI 10.1.2.31@tcp [ 711.406533] Key type .llcrypt unregistered [ 711.408759] Key type ._llcrypt unregistered [ 722.290657] Key type ._llcrypt registered [ 722.292378] Key type .llcrypt registered [ 723.502238] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:51:55 (1764733915) [ 723.835878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.848455] alg: No test for adler32 (adler32-zlib) [ 724.792765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.858740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.889575] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 725.770063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.830906] LNet: 38653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.853035] LNetError: 38653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.892539] LNet: Removed LNI 192.168.204.9@tcp [ 725.954683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.072855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.9@tcp [ 726.138326] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 726.974393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.9@tcp [ 727.003504] LNet: Removed LNI 192.168.204.9@tcp [ 727.049472] LNet: 38788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.639469] Key type .llcrypt unregistered [ 727.643227] Key type ._llcrypt unregistered [ 740.380982] Key type ._llcrypt registered [ 740.382722] Key type .llcrypt registered [ 742.335752] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:52:13 (1764733933) [ 745.121325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg409-server [ 747.284688] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:52:19 (1764733939) [ 748.140305] Key type .llcrypt unregistered [ 748.141870] Key type ._llcrypt unregistered [ 755.638760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.799209] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.806047] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.808541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.433648] Key type ._llcrypt registered [ 758.434947] Key type .llcrypt registered [ 758.766187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 758.803823] alg: No test for adler32 (adler32-zlib) [ 759.784959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.844448] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 760.911946] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 763.631803] LNet: 40614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.653359] LNetError: 40614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.670115] LNet: Removed LNI 192.168.204.9@tcp1 [ 763.681707] LNet: Removed LNI 10.1.2.3@tcp2 [ 764.366491] Key type .llcrypt unregistered [ 764.372239] Key type ._llcrypt unregistered [ 774.121535] Key type ._llcrypt registered [ 774.129428] Key type .llcrypt registered [ 775.220719] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:52:47 (1764733967) [ 775.555223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.570358] alg: No test for adler32 (adler32-zlib) [ 776.466491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.521444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.538708] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 777.198853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.216507] LNet: 41578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.220973] LNetError: 41578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.234307] LNet: Removed LNI 192.168.204.9@tcp [ 777.254073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.306399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 777.335175] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 782.756701] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:52:55 (1764733975) [ 782.927284] LNet: 42170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.934782] LNetError: 42170:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.944748] LNet: Removed LNI 192.168.204.9@tcp [ 783.489211] Key type .llcrypt unregistered [ 783.493262] Key type ._llcrypt unregistered [ 787.877347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 788.022526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 788.025248] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 788.040456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.663396] Key type ._llcrypt registered [ 790.664860] Key type .llcrypt registered [ 790.823629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.843364] alg: No test for adler32 (adler32-zlib) [ 791.741064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.789943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.827767] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 792.529686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.547691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.473312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.496344] LNet: 42910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.502572] LNetError: 42910:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.509492] LNet: Removed LNI 192.168.204.9@tcp [ 794.523124] LNet: Removed LNI 10.1.2.3@tcp [ 794.547741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.625650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 794.667591] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 794.680753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.202731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.221329] LNet: 42937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.228181] LNetError: 42937:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.231746] LNetError: 42937:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 795.236875] LNet: Removed LNI 192.168.204.9@tcp [ 795.250642] LNet: Removed LNI 10.1.2.3@tcp [ 795.269528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.330532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 795.379169] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 795.383662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.720497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 795.750808] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 796.656987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.693638] LNet: 43099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.698091] LNetError: 43099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.701732] LNetError: 43099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 796.706847] LNet: Removed LNI 192.168.204.9@tcp [ 796.710178] LNet: Removed LNI 10.1.2.3@tcp [ 796.717854] LNet: Removed LNI 192.168.204.9@tcp2 [ 796.751480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.811816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 796.840889] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 796.845428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 796.850295] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 803.915809] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:53:16 (1764733996) [ 804.104614] LNet: 43696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.109462] LNetError: 43696:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.114737] LNetError: 43696:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 804.120816] LNet: Removed LNI 192.168.204.9@tcp [ 804.124236] LNet: Removed LNI 10.1.2.3@tcp [ 804.134947] LNet: Removed LNI 192.168.204.9@tcp2 [ 804.744524] Key type .llcrypt unregistered [ 804.746159] Key type ._llcrypt unregistered [ 808.955127] Key type ._llcrypt registered [ 808.956602] Key type .llcrypt registered [ 809.219963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.237484] alg: No test for adler32 (adler32-zlib) [ 810.213844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.293098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.320204] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 811.095728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.118191] LNet: 44083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.124497] LNetError: 44083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.138336] LNet: Removed LNI 192.168.204.9@tcp [ 811.170311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.263698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 811.312325] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 811.772112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.806580] LNet: 44113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.814483] LNetError: 44113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.838529] LNet: Removed LNI 192.168.204.9@tcp [ 811.863795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.945553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 811.972537] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 812.449508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.497923] LNet: Removed LNI 192.168.204.9@tcp [ 812.535556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.599738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 812.634171] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 813.153697] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.249969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.252829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.255474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.766978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.796369] LNet: 44222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.800772] LNet: 44222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 815.826273] LNetError: 44222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.829633] LNetError: 44222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 815.854502] LNet: Removed LNI 192.168.204.9@tcp [ 815.879459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.971255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.015889] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 821.661844] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:53:33 (1764734013) [ 821.837916] LNet: 44810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.846148] LNetError: 44810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.904091] LNet: Removed LNI 192.168.204.9@tcp [ 822.476919] Key type .llcrypt unregistered [ 822.482228] Key type ._llcrypt unregistered [ 826.982381] Key type ._llcrypt registered [ 826.988668] Key type .llcrypt registered [ 827.201578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.216241] alg: No test for adler32 (adler32-zlib) [ 828.149536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.195535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.222185] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 829.065386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.087722] LNet: 45213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.093945] LNetError: 45213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.110269] LNet: Removed LNI 192.168.204.9@tcp [ 829.126497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.188545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 829.219521] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 835.043913] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:53:47 (1764734027) [ 835.182605] LNet: 45801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.193536] LNetError: 45801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.212637] LNet: Removed LNI 192.168.204.9@tcp [ 835.830576] Key type .llcrypt unregistered [ 835.832395] Key type ._llcrypt unregistered [ 840.222105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.272650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.279105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.284319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.874602] Key type ._llcrypt registered [ 842.876305] Key type .llcrypt registered [ 843.048587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.070226] alg: No test for adler32 (adler32-zlib) [ 843.964465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.018424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 844.048445] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 844.795587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.810838] LNet: 46247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.814967] LNetError: 46247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.827936] LNet: Removed LNI 192.168.204.9@tcp [ 844.844068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.889206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 844.909739] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 850.367545] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:54:02 (1764734042) [ 850.423614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.459535] LNet: 46838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.465289] LNetError: 46838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.486642] LNet: Removed LNI 192.168.204.9@tcp [ 850.504386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.657825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 855.354150] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:54:07 (1764734047) [ 855.538510] LNet: 47427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.020892] Key type .llcrypt unregistered [ 856.022549] Key type ._llcrypt unregistered [ 860.478226] Key type ._llcrypt registered [ 860.479612] Key type .llcrypt registered [ 860.719349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.751545] alg: No test for adler32 (adler32-zlib) [ 861.677612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.725380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.770873] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 862.714326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.735227] LNet: 47821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.741154] LNetError: 47821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.751138] LNet: Removed LNI 192.168.204.9@tcp [ 862.788654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.854605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 862.898863] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 868.776109] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:54:21 (1764734061) [ 868.906535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.931932] LNet: 48414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.941729] LNetError: 48414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.950634] LNet: Removed LNI 192.168.204.9@tcp [ 868.969095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.033903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 874.233860] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:54:26 (1764734066) [ 875.462784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 876.739660] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:54:29 (1764734069) [ 877.951579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 879.357391] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:54:31 (1764734071) [ 880.826968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 882.493174] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:54:34 (1764734074) [ 883.845884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 885.212381] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:54:37 (1764734077) [ 886.544655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 887.910136] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:54:40 (1764734080) [ 888.121325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.154387] LNet: 50465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.199357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.265132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.284976] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 888.976867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.9 [ 888.998360] LNet: 50599:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 889.005222] LNet: Removed LNI 192.168.204.9@tcp [ 894.308188] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:54:46 (1764734086) [ 894.361930] LNet: 51162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.368201] LNetError: 51162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.380700] LNet: Removed LNI 192.168.204.9@tcp [ 895.068701] Key type .llcrypt unregistered [ 895.070541] Key type ._llcrypt unregistered [ 899.366847] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.580453] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.582766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.584930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.187435] Key type ._llcrypt registered [ 902.188972] Key type .llcrypt registered [ 902.335482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.345898] alg: No test for adler32 (adler32-zlib) [ 903.576712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.026141] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:55:01 (1764734101) [ 909.216368] LNet: 52070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.224653] LNetError: 52070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.242371] LNet: Removed LNI 10.1.2.3@tcp [ 909.769449] Key type .llcrypt unregistered [ 909.771183] Key type ._llcrypt unregistered [ 914.331793] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.564552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.579716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.582499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.211496] Key type ._llcrypt registered [ 917.213457] Key type .llcrypt registered [ 917.367835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.383932] alg: No test for adler32 (adler32-zlib) [ 918.311972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.699319] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:55:15 (1764734115) [ 923.938503] LNet: 52959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.942310] LNetError: 52959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 923.960378] LNet: Removed LNI 10.1.2.3@tcp [ 924.509529] Key type .llcrypt unregistered [ 924.512440] Key type ._llcrypt unregistered [ 929.489484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 929.784675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 929.787261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 929.799957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 932.527266] Key type ._llcrypt registered [ 932.532565] Key type .llcrypt registered [ 932.716637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.739181] alg: No test for adler32 (adler32-zlib) [ 933.742714] LNet: 53256:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 933.761240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 939.618888] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:55:31 (1764734131) [ 939.779126] LNet: 53839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.787774] LNetError: 53839:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 939.821810] LNet: Removed LNI 10.1.2.3@tcp [ 940.361757] Key type .llcrypt unregistered [ 940.364797] Key type ._llcrypt unregistered [ 945.192897] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.411492] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.413423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.421150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.100487] Key type ._llcrypt registered [ 948.105072] Key type .llcrypt registered [ 948.301346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.314361] alg: No test for adler32 (adler32-zlib) [ 949.209176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.268667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 956.186738] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:55:48 (1764734148) [ 956.593179] LNet: 55072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.608232] LNetError: 55072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.627520] LNet: Removed LNI 10.1.2.3@tcp [ 957.240973] Key type .llcrypt unregistered [ 957.246384] Key type ._llcrypt unregistered [ 965.172139] Key type ._llcrypt registered [ 965.173824] Key type .llcrypt registered [ 965.566959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 965.614267] alg: No test for adler32 (adler32-zlib) [ 966.758933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 966.797858] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 979.490784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 980.780866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 980.916975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 981.001075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 981.067508] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 981.161717] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 981.618846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.714032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 981.754822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.823320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.888222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.989363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 982.075975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 982.181825] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 982.187652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 982.253521] LNet: 55508:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.279124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 982.375730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 982.465283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 982.547751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 982.642239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.740620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.816247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 983.882793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 983.994418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 984.104711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 984.170527] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 984.661904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.744990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 984.815277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 984.853187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 984.942979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.020685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.093939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.147916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.183249] LNet: 55618:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.222703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.273529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.307429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.391262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.478426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.523222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.600980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 986.482072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 986.543470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 986.603723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 986.639249] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 987.092536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.162797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 987.222418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 987.269766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 987.385039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 987.491342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 987.595150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 987.682882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.741297] LNet: 55727:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.774296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.812206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.860582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.931730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.973243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.048271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.106915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 989.039986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 989.118733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 989.184339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 989.531518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.604952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 989.711084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.813546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.897919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.010204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.089352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.151949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.183514] LNet: 55837:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.241474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.326168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.385910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.441674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.485499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.531689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.570197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 991.335303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 991.376581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 991.434869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 991.464853] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 991.476488] LNet: Skipped 1 previous similar message [ 991.850032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.902690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 991.942278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.979550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.043658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.125862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.172424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.245743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.300648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.349484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.423785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.514711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.587130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.677280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.801201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 993.922748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 994.009592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 994.106514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 994.515927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.561783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 994.623106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.690917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.763567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.828239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.869596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.932860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.987932] LNet: 56056:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 995.021067] LNet: 56056:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 995.068155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.150834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.243459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.321141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.427526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.539420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.627212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 996.129399] LNet: 56096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.153919] LNetError: 56096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.198916] LNet: Removed LNI 192.168.204.9@tcp [ 996.881554] Key type .llcrypt unregistered [ 996.883125] Key type ._llcrypt unregistered [ 1015.641345] Key type ._llcrypt registered [ 1015.642593] Key type .llcrypt registered [ 1016.943254] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:56:48 (1764734208) [ 1017.780422] Key type .llcrypt unregistered [ 1017.782136] Key type ._llcrypt unregistered [ 1023.469825] Key type ._llcrypt registered [ 1023.471530] Key type .llcrypt registered [ 1023.805904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.840202] alg: No test for adler32 (adler32-zlib) [ 1024.824034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1024.880550] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1031.595955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1040.712686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.775233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1040.796653] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1042.873166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1042.909094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1042.964451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1043.011364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1043.078058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1043.123564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1043.177252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1043.240958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1043.303234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1043.323853] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1043.328087] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1043.519476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.563095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.601291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.654325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.690630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.716392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.761328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.799961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.825828] LNet: 57558:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.856703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.879420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.906707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.934645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.961152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.991562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.021735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1044.551300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1044.570893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1044.595561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1044.627197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1044.651233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1044.676725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1044.705818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1044.730224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1044.768571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1044.785220] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1044.791118] LNet: Skipped 2 previous similar messages [ 1045.001963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.023885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.072435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.099621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.130311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.170882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.215572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.253062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.267152] LNet: 57680:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1045.299604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.339141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.388201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.426411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.497765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.566914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.643846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1046.253702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1046.334607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1046.373071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1046.412320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1046.458233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1046.514627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1046.558777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1046.595725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1046.643491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1046.660643] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1046.665761] LNet: Skipped 2 previous similar messages [ 1046.894451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.934966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.959281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.988707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.047841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.091793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.124966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.159991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.176567] LNet: 57801:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.192969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.222709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.249200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.302978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.328150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.347482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.381763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.936768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1047.995483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1048.023791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1048.073423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1048.118690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1048.153901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1048.187949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1048.239476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1048.289996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1048.539665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.579119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.617921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.640979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.664883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.707707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.755910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.802127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.835552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.870783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.889517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.925983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.957794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.977206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.016917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1049.510814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1049.555569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1049.587882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1049.614715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1049.687960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1049.763488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1049.807361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1049.861230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1049.893606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1049.912623] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1049.916362] LNet: Skipped 5 previous similar messages [ 1050.156675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.190095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1050.240929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.267453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.308375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.341626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.369983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.397129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.414750] LNet: 58044:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.432249] LNet: 58044:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1050.449442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.480865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.511484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.544318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.584473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.621908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.674344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1051.233367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1051.277859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1051.320393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1051.351189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1051.378452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1051.411479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1051.458889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1051.489505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1051.528455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1051.794796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.816454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1051.839405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.875812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.905440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.930197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.960896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.984677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.028632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.049917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.080595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.108512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.149721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.177820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.207842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.926889] LNet: 58240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.936040] LNetError: 58240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.944619] LNet: Removed LNI 192.168.204.9@tcp [ 1057.963731] LNet: Removed LNI 192.168.204.9@tcp1 [ 1058.404830] Key type .llcrypt unregistered [ 1058.409145] Key type ._llcrypt unregistered [ 1071.491935] Key type ._llcrypt registered [ 1071.494915] Key type .llcrypt registered [ 1072.587031] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:57:44 (1764734264) [ 1073.292873] Key type .llcrypt unregistered [ 1073.296515] Key type ._llcrypt unregistered [ 1077.619771] Key type ._llcrypt registered [ 1077.621628] Key type .llcrypt registered [ 1077.807993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.818817] alg: No test for adler32 (adler32-zlib) [ 1078.674205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1078.704439] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1084.176661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1084.780335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1084.800185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1084.823523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1084.841793] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1084.980471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.007887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.043860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.075696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.096423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.120419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.143847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.166967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.197409] LNet: 59481:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.221886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.243874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.268594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.287959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.326293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.386590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.422603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1085.814482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1085.839188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1085.865272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1085.886890] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1086.050102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.084920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1086.106788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.157492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.188346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.236477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.272903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.307170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.324214] LNet: 59591:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.338604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.369538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.396881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.429483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.454489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.477643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.512094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.954615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1086.984679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1087.014792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1087.039960] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1087.255260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.278105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1087.301878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.326646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.351132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.383608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.418365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.446038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.457123] LNet: 59700:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.477152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.505365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.548587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.579105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.602967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.639474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.668616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1087.901214] LNet: 59740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.908435] LNetError: 59740:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.938811] LNet: Removed LNI 192.168.204.9@tcp [ 1089.338915] Key type .llcrypt unregistered [ 1089.345761] Key type ._llcrypt unregistered [ 1104.339642] Key type ._llcrypt registered [ 1104.349222] Key type .llcrypt registered [ 1105.638882] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:58:17 (1764734297) [ 1106.302500] Key type .llcrypt unregistered [ 1106.304344] Key type ._llcrypt unregistered [ 1110.688484] Key type ._llcrypt registered [ 1110.690173] Key type .llcrypt registered [ 1110.859137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.876876] alg: No test for adler32 (adler32-zlib) [ 1111.759593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1111.797246] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1117.312320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1123.735347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1123.764464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1123.782480] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1124.976524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1125.011685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1125.042718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1125.069216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1125.107636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1125.146424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1125.182076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1125.215549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1125.264886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1125.297666] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1125.304330] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1125.517394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.567382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1125.624444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.671445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.708740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.759384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.810137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.853371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.872250] LNet: 61202:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.884106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.913628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.962578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.001923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.042194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.083515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.120248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.159585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.601461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1126.628707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1126.657907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1126.706636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1126.730287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1126.769913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1126.821598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1126.860831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1126.921842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1126.937826] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1126.941694] LNet: Skipped 2 previous similar messages [ 1127.154054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.202959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1127.246272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.276418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.313623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.358718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.397110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.422700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.438593] LNet: 61326:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.451291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.487324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.525584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.549518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.585240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.615537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.670827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.695589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.243325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1128.278081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1128.315648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1128.345386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1128.382916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1128.411864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1128.433090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1128.466164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1128.514699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1128.532800] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1128.541787] LNet: Skipped 1 previous similar message [ 1128.703721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.743115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1128.774490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.821420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.872935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.913517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.953329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.977852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.997556] LNet: 61449:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.031602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.065849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.103931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.140307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.164804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.197434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.223283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.258211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.475743] LNet: 61526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.489150] LNetError: 61526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.492796] LNet: Removed LNI 192.168.204.9@tcp [ 1133.515241] LNet: Removed LNI 192.168.204.9@tcp1 [ 1133.886471] Key type .llcrypt unregistered [ 1133.887692] Key type ._llcrypt unregistered [ 1146.808316] Key type ._llcrypt registered [ 1146.810520] Key type .llcrypt registered [ 1147.842877] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:59:00 (1764734340) [ 1148.376332] Key type .llcrypt unregistered [ 1148.378093] Key type ._llcrypt unregistered [ 1153.205220] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1153.297883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1153.303242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1153.309261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1155.743767] Key type ._llcrypt registered [ 1155.746314] Key type .llcrypt registered [ 1155.873309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1155.881934] alg: No test for adler32 (adler32-zlib) [ 1156.793161] LNet: 62598:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1156.812225] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1156.836921] LNet: 62612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.841573] LNetError: 62612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.853229] LNet: Removed LNI 192.168.204.9@tcp [ 1157.435394] Key type .llcrypt unregistered [ 1157.436685] Key type ._llcrypt unregistered [ 1162.308740] Key type ._llcrypt registered [ 1162.310177] Key type .llcrypt registered [ 1162.521878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.543793] alg: No test for adler32 (adler32-zlib) [ 1163.413031] LNet: 62867:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1163.431569] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1163.433729] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1163.465405] LNet: 62881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.470335] LNetError: 62881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.477095] LNet: Removed LNI 192.168.204.9@tcp [ 1163.484465] LNet: Removed LNI 10.1.2.3@tcp1 [ 1164.049468] Key type .llcrypt unregistered [ 1164.051929] Key type ._llcrypt unregistered [ 1167.770630] Key type ._llcrypt registered [ 1167.772313] Key type .llcrypt registered [ 1167.903916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.923427] alg: No test for adler32 (adler32-zlib) [ 1168.813611] LNet: 63136:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1168.827713] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1168.829782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1168.871961] LNet: 63151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.877697] LNetError: 63151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.885804] LNet: Removed LNI 192.168.204.9@tcp [ 1168.901516] LNet: Removed LNI 10.1.2.3@tcp [ 1169.406171] Key type .llcrypt unregistered [ 1169.407723] Key type ._llcrypt unregistered [ 1174.110351] Key type ._llcrypt registered [ 1174.112781] Key type .llcrypt registered [ 1174.290739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1174.301857] alg: No test for adler32 (adler32-zlib) [ 1175.226972] LNet: 63407:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1175.239641] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1175.242338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1175.279937] LNet: 63421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.283476] LNetError: 63421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.287626] LNet: Removed LNI 192.168.204.9@tcp [ 1175.295167] LNet: Removed LNI 10.1.2.3@tcp [ 1175.884374] Key type .llcrypt unregistered [ 1175.887622] Key type ._llcrypt unregistered [ 1180.761212] Key type ._llcrypt registered [ 1180.762811] Key type .llcrypt registered [ 1180.944946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.968225] alg: No test for adler32 (adler32-zlib) [ 1181.833327] LNet: 63681:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1181.854626] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1181.856801] LNetError: 63681:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1181.861670] LNetError: Error -100 starting up LNI tcp [ 1181.878779] LNet: Removed LNI 192.168.204.9@tcp [ 1182.440417] Key type .llcrypt unregistered [ 1182.443744] Key type ._llcrypt unregistered [ 1191.527484] Key type ._llcrypt registered [ 1191.529112] Key type .llcrypt registered [ 1192.765909] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:59:44 (1764734384) [ 1193.527588] Key type .llcrypt unregistered [ 1193.529877] Key type ._llcrypt unregistered [ 1198.170183] Key type ._llcrypt registered [ 1198.171751] Key type .llcrypt registered [ 1198.364480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.380985] alg: No test for adler32 (adler32-zlib) [ 1199.252285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.274142] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1205.980824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1206.660765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1206.696862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1206.730161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1206.751510] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1206.911514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.937844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1206.962756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.986512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.039411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.095542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.136561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.169570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.192315] LNet: 64930:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.213798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.255514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.310272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.351451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.403548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.438211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.643295] LNet: 64968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.650992] LNetError: 64968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.677211] LNet: Removed LNI 192.168.204.9@tcp [ 1208.070537] Key type .llcrypt unregistered [ 1208.072153] Key type ._llcrypt unregistered [ 1221.902418] Key type ._llcrypt registered [ 1221.904186] Key type .llcrypt registered [ 1222.161860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.194505] alg: No test for adler32 (adler32-zlib) [ 1223.155772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.183443] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1230.232377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1237.616100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.658510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1237.674888] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1239.022584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1239.048522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1239.082744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1239.113026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1239.145842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1239.190410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1239.225561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1239.256610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1239.283170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1239.305603] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1239.491379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.533172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1239.575553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.584428] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1239.586756] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1239.619589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.645145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.680331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.701208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.723892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.740724] LNet: 65840:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.759677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.785500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.808159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.831862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.880679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.931372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.559703] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1243.968987] LNet: 65913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.979847] LNetError: 65913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.029799] LNet: Removed LNI 192.168.204.9@tcp [ 1245.031588] LNetError: 65913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.054487] LNet: Removed LNI 192.168.204.9@tcp1 [ 1245.444710] Key type .llcrypt unregistered [ 1245.453427] Key type ._llcrypt unregistered [ 1258.505429] Key type ._llcrypt registered [ 1258.507064] Key type .llcrypt registered [ 1259.851397] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:00:51 (1764734451) [ 1260.200754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.208238] alg: No test for adler32 (adler32-zlib) [ 1261.165563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.228955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1261.261517] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1261.902357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1261.918041] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1262.717720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1262.777979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1262.827590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1262.868427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1262.903637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1262.927736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1262.963312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1262.983611] LNetError: 67010:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1263.266412] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1263.270224] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1265.312782] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1266.477308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1269.408658] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1270.114282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1270.150293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.214916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.252070] LNet: 67130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.257113] LNetError: 67130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.267255] LNet: Removed LNI 192.168.204.9@tcp [ 1270.273776] LNet: Removed LNI 192.168.204.9@tcp1 [ 1270.290340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.340590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.375275] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1271.191173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.218337] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1272.305871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1272.362629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.448557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1272.486816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1272.528724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1272.572718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1272.628111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1272.681993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1272.706054] LNetError: 67403:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1273.379488] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1276.255784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1279.520402] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1279.523368] LNet: Skipped 1 previous similar message [ 1284.169955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1284.216749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1284.246601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.906379] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:01:22 (1764734482) [ 1290.237947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.253753] LNet: 68120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.262899] LNetError: 68120:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.265993] LNetError: 68120:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1290.269617] LNet: Removed LNI 192.168.204.9@tcp [ 1290.281769] LNet: Removed LNI 192.168.204.9@tcp1 [ 1290.307837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.362336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1290.390365] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1291.001711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1291.024366] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1291.853803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1291.934497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1291.971618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1292.009292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1292.054547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1292.104040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1292.132803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 1292.204332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1292.385088] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.9@tcp [ 1293.411943] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1293.418726] LNet: Skipped 1 previous similar message [ 1308.360253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.553404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.591775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.643397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 500 [ 1308.726068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.785224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.840583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1308.855920] LNet: 68571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.861731] LNetError: 68571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.866754] LNetError: 68571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1308.870441] LNet: Removed LNI 192.168.204.9@tcp [ 1308.882110] LNet: Removed LNI 192.168.204.9@tcp1 [ 1308.896555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.951661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.977076] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1309.576961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1309.591813] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1310.294941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1310.326849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1310.407389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1310.451569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1310.481386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1310.518461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1310.544631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1310.575120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 1311.968393] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1311.974380] LNet: Skipped 3 previous similar messages [ 1312.929851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.929485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.951426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1322.991698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.722799] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:02:00 (1764734520) [ 1328.123982] LNet: 69626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.132738] LNetError: 69626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.139380] LNetError: 69626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1328.143844] LNet: Removed LNI 192.168.204.9@tcp [ 1328.151791] LNet: Removed LNI 192.168.204.9@tcp1 [ 1328.715919] Key type .llcrypt unregistered [ 1328.724173] Key type ._llcrypt unregistered [ 1332.560382] Key type ._llcrypt registered [ 1332.562134] Key type .llcrypt registered [ 1332.727935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.736392] alg: No test for adler32 (adler32-zlib) [ 1333.590193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1333.613770] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1340.415279] Lustre: DEBUG MARKER: Initial discovery [ 1340.455674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1341.904650] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1341.988392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1342.023283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1342.040384] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1342.054529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1342.074470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -l 3 [ 1342.103374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -e local_error [ 1342.132213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m PUT -l 6 [ 1342.880102] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1342.979565] Lustre: DEBUG MARKER: Force oleg409-client.virtnet to discover 192.168.204.109@tcp (in background) [ 1343.011427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1343.808630] Lustre: DEBUG MARKER: Force oleg409-server to discover 192.168.204.9@tcp [ 1345.368181] Lustre: DEBUG MARKER: Wait for 70375 [ 1346.080386] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1347.013741] Lustre: DEBUG MARKER: Finished wait on 70375 [ 1347.053161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1347.079716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1347.259991] LNet: 70833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.263827] LNetError: 70833:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.279171] LNet: Removed LNI 192.168.204.9@tcp [ 1347.568625] Key type .llcrypt unregistered [ 1347.570349] Key type ._llcrypt unregistered [ 1356.071646] Key type ._llcrypt registered [ 1356.073806] Key type .llcrypt registered [ 1356.945484] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:02:29 (1764734549) [ 1357.233301] Key type .llcrypt unregistered [ 1357.235061] Key type ._llcrypt unregistered [ 1360.065850] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1360.180444] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1360.186489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1360.189853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1362.488511] Key type ._llcrypt registered [ 1362.490203] Key type .llcrypt registered [ 1362.614880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1362.626206] alg: No test for adler32 (adler32-zlib) [ 1363.481707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.522297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.546592] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1364.036385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1364.051183] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1368.970871] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:02:41 (1764734561) [ 1369.019873] LNet: 72893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.025763] LNetError: 72893:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.033496] LNet: Removed LNI 192.168.204.9@tcp [ 1369.045566] LNet: Removed LNI 10.1.2.3@tcp [ 1369.432839] Key type .llcrypt unregistered [ 1369.434610] Key type ._llcrypt unregistered [ 1372.256989] Key type ._llcrypt registered [ 1372.258679] Key type .llcrypt registered [ 1372.340313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1372.347465] alg: No test for adler32 (adler32-zlib) [ 1373.185780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.215566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.229521] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1373.636778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1373.648818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1374.145437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.9@tcp [ 1375.200244] LNetError: Unexpected error -22 connecting to 192.168.204.9@tcp at host 192.168.204.9:988 [ 1377.634794] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:02:50 (1764734570) [ 1377.684424] LNet: 74020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.688755] LNetError: 74020:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.692186] LNet: Removed LNI 192.168.204.9@tcp [ 1377.701650] LNet: Removed LNI 10.1.2.3@tcp [ 1378.051654] Key type .llcrypt unregistered [ 1378.053122] Key type ._llcrypt unregistered [ 1380.597770] Key type ._llcrypt registered [ 1380.599630] Key type .llcrypt registered [ 1380.695741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.706242] alg: No test for adler32 (adler32-zlib) [ 1381.525389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.559867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.574265] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1381.964258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1381.978441] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1382.418384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.9@tcp1 --nid 192.168.204.9@tcp2 [ 1384.553098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp2 [ 1384.564865] LNetError: 74857:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.9@tcp1 to 192.168.204.9@tcp2 - Route table may be misconfigured [ 1384.577393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp2 192.168.204.9@tcp1 [ 1388.041556] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:03:00 (1764734580) [ 1388.105322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.124343] LNet: 75423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.128521] LNetError: 75423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.132126] LNet: Removed LNI 192.168.204.9@tcp1 [ 1389.160192] LNet: Removed LNI 192.168.204.9@tcp2 [ 1389.188735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.246470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.264429] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1389.704417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.719920] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1390.211274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1390.239184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.9@tcp [ 1390.263994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp1 [ 1390.287702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp -e network_timeout [ 1390.308205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -e network_timeout [ 1390.330498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -e network_timeout [ 1390.358730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -e network_timeout [ 1390.380711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1390.394130] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1390.408543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1390.471767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.525152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1391.264705] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1394.085378] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:03:06 (1764734586) [ 1394.136477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.149144] LNet: 76284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.153648] LNetError: 76284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.157104] LNetError: 76284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1394.161221] LNet: Removed LNI 192.168.204.9@tcp [ 1395.180903] LNet: Removed LNI 192.168.204.9@tcp1 [ 1395.198958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.253134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1395.428055] LNet: 76327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.670894] Key type .llcrypt unregistered [ 1395.672746] Key type ._llcrypt unregistered [ 1401.092900] Key type ._llcrypt registered [ 1401.094434] Key type .llcrypt registered [ 1401.746622] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:03:14 (1764734594) [ 1401.927303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.945671] alg: No test for adler32 (adler32-zlib) [ 1402.772646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.631927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1404.538957] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:03:17 (1764734597) [ 1404.653401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.667231] LNet: 77420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.678865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.706442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1404.721194] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1405.109095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1405.121591] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1405.558873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1405.577710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp1 [ 1405.601593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp1 [ 1408.998407] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:03:21 (1764734601) [ 1410.059089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg409-server [ 1411.150875] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:03:23 (1764734603) [ 1412.386879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg409-server [ 1413.474259] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:03:25 (1764734605) [ 1414.513690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg409-server [ 1415.341572] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:03:28 (1764734608) [ 1416.358272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg409-server [ 1417.243219] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:03:29 (1764734609) [ 1418.158729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg409-server [ 1419.009956] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:03:31 (1764734611) [ 1419.823561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg409-server [ 1420.582450] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:03:33 (1764734613) [ 1421.485630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg409-server [ 1422.293665] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:03:35 (1764734615) [ 1423.288752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg409-server [ 1424.132347] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:03:36 (1764734616) [ 1425.055846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg409-server [ 1425.861434] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:03:38 (1764734618) [ 1425.921496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.935615] LNet: 81292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.939981] LNetError: 81292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.943200] LNet: Removed LNI 192.168.204.9@tcp [ 1426.986178] LNet: Removed LNI 192.168.204.9@tcp1 [ 1427.001566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.037454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1427.058631] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1427.470961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1427.512880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1427.613783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.627172] LNet: 81442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.630669] LNetError: 81442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.633711] LNetError: 81442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1427.648327] LNet: Removed LNI 192.168.204.9@tcp [ 1427.661037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.694249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1427.712352] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1428.221932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1428.352654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.415690] LNet: Removed LNI 192.168.204.9@tcp [ 1429.434172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.485262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.510853] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1429.918121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1429.984829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1430.083425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.094492] LNet: 81734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.097943] LNet: 81734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1430.101942] LNetError: 81734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.105852] LNetError: 81734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1430.115490] LNet: Removed LNI 192.168.204.9@tcp [ 1430.125711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.151543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1430.170915] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1430.554546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1430.656583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.718861] LNet: Removed LNI 192.168.204.9@tcp [ 1431.732404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.769926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.782899] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1432.163042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1432.209541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1432.320852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1432.332374] LNetError: 82027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.335765] LNetError: 82027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1432.343601] LNet: Removed LNI 192.168.204.9@tcp [ 1432.358120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.387693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1432.403831] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1432.786494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1432.919377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.963620] LNet: Removed LNI 192.168.204.9@tcp [ 1433.981223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.027506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.044773] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1434.473052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1434.514068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1434.656755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.671598] LNet: 82319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.675861] LNet: 82319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1435.751082] LNet: Removed LNI 192.168.204.9@tcp [ 1435.766303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.812615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1435.827704] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1436.238274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1436.390329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.404604] LNetError: 82464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.408313] LNetError: 82464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1436.416574] LNet: Removed LNI 192.168.204.9@tcp [ 1436.428977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.460774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1436.475228] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1436.855191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1436.894214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1437.025517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.054662] LNet: Removed LNI 192.168.204.9@tcp [ 1438.067977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.103132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1438.124224] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1438.581303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1439.857673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.935336] LNet: Removed LNI 192.168.204.9@tcp [ 1440.956573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.995481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1441.012169] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1441.518711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1441.567429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1441.796669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.855730] LNet: Removed LNI 192.168.204.9@tcp [ 1442.873078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.930517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1442.956030] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1443.461931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1444.717263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.730757] LNet: 83068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.734306] LNet: 83068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1444.739072] LNetError: 83068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.743527] LNetError: 83068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1444.755897] LNet: Removed LNI 192.168.204.9@tcp [ 1444.770625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.802789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.817842] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1445.193945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1445.222933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.248729] LNet: Removed LNI 192.168.204.9@tcp [ 1445.261274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.297037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.311543] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1445.736928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1449.480743] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:04:02 (1764734642) [ 1449.524069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.543771] LNet: Removed LNI 192.168.204.9@tcp [ 1449.553629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.583262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1449.597503] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1450.137439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.154607] LNet: Removed LNI 192.168.204.9@tcp231 [ 1450.164271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.196982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1450.212153] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1450.215625] LNet: 84051:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1450.406952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1450.423244] LNet: Removed LNI 192.168.204.9@tcp231 [ 1450.432715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1450.447104] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1450.988132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.005771] LNet: Removed LNI 192.168.204.9@tcp231 [ 1451.015526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.050116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1451.065132] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 1451.069435] LNet: 84203:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.290494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1451.312149] LNet: Removed LNI 192.168.204.9@tcp231 [ 1451.320546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1451.338858] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 1451.859478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.883690] LNet: Removed LNI 192.168.204.9@tcp231 [ 1451.893366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.933719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1451.949657] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 1452.131385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1452.154879] LNet: Removed LNI 192.168.204.9@tcp231 [ 1452.164489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1452.180211] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 1452.721574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.741690] LNet: Removed LNI 192.168.204.9@tcp231 [ 1452.753362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.793328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1452.807906] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1452.811960] LNet: 84505:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1452.815605] LNet: 84505:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1453.008090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1453.024369] LNet: Removed LNI 192.168.204.9@tcp231 [ 1453.033550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1453.050243] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1456.432500] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:04:09 (1764734649) [ 1456.487896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.511544] LNet: Removed LNI 192.168.204.9@tcp231 [ 1456.526879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.565989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1456.580635] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1456.997150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1457.019241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1457.036849] LNet: Removed LNI 192.168.204.9@tcp232 [ 1457.047947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1457.067031] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1457.479365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1457.500197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1457.525214] LNet: Removed LNI 192.168.204.9@tcp232 [ 1457.539631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1457.561369] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1457.959364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1457.981650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1458.001709] LNet: Removed LNI 192.168.204.9@tcp232 [ 1458.012769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1458.028753] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1458.445655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1458.475959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1458.498316] LNet: Removed LNI 192.168.204.9@tcp232 [ 1458.509265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1461.754534] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:04:14 (1764734654) [ 1462.005035] LNet: 86344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.009211] LNet: 86344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1462.356307] Key type .llcrypt unregistered [ 1462.357985] Key type ._llcrypt unregistered [ 1465.158789] Key type ._llcrypt registered [ 1465.161426] Key type .llcrypt registered [ 1465.275843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1465.283374] alg: No test for adler32 (adler32-zlib) [ 1466.105069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1466.122799] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1469.752894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1473.344351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.365523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.382667] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1474.110000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 1474.132496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 1474.151910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 1474.175440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 1474.199153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 1474.222866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 1474.254040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 1474.273350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 1474.295623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1474.307855] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1474.313780] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1474.435719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1474.455854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1474.473921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1474.491431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1474.509722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1474.529934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1474.550658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1474.568471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1474.579613] LNet: 87016:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1474.589403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1474.605796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1474.623912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1474.641861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1474.661445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1474.681974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1474.956633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 1474.977518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 1474.999524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 1475.019595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 1475.041941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 1475.068759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 1475.088215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 1475.107896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 1475.132079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1475.147675] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1475.277271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1475.294569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1475.310855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1475.328062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1475.345507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1475.363430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1475.382081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1475.406713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1475.415560] LNet: 87136:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1475.426967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1475.444370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1475.463054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1475.481531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1475.501058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1475.521295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1475.776126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 1475.799706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 1475.823536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 1475.843305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 1475.865193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 1475.884225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 1475.905488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 1475.926738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 1475.950811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1476.095483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1476.113285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1476.133671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1476.154816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1476.178970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1476.205844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1476.234306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1476.262267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1476.280285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1476.300518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1476.326221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1476.346950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.366991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1476.386208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1476.662440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 1476.685453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 1476.711628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 1476.733914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 1476.754507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 1476.780441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 1476.804039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 1476.833571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 1476.853632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1476.871734] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1476.875617] LNet: Skipped 2 previous similar messages [ 1476.992540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1477.009902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1477.028734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1477.047496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1477.064953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1477.085737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1477.105149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1477.121452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1477.131208] LNet: 87377:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1477.134622] LNet: 87377:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1477.143886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1477.161385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1477.177513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1477.198101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1477.219491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1477.242228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1477.545725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 1477.566836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 1477.588275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 1477.618938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 1477.643679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 1477.667540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 1477.692531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 1477.716826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 1477.736765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1477.864576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1477.883355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1477.902744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1477.923939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1477.944631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1477.962984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1477.985988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1478.002886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1478.017961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1478.036124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1478.051773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1478.069709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.086566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1478.103838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1478.330588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 1478.349629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 1478.370623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 1478.390518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 1478.411242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 1478.430288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 1478.449802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 1478.469060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 1478.488434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1478.501712] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1478.601177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1478.616524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1478.632465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1478.649717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1478.666515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1478.682413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1478.699633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1478.717779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1478.736307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1478.752548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1478.769359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1478.785680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.803108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1478.820309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1481.310799] LNet: 87690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.314495] LNetError: 87690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.318529] LNet: Removed LNI 192.168.204.9@tcp [ 1481.332496] LNet: Removed LNI 192.168.204.9@tcp1 [ 1481.519911] Key type .llcrypt unregistered [ 1481.521745] Key type ._llcrypt unregistered [ 1488.605243] Key type ._llcrypt registered [ 1488.606820] Key type .llcrypt registered [ 1489.371212] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:04:41 (1764734681) [ 1489.562216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.574539] alg: No test for adler32 (adler32-zlib) [ 1490.419860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.457823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.481066] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1490.936179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1490.958840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1490.970425] LNetError: 88663:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1491.017190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.9@tcp [ 1491.083570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1494.242471] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:04:46 (1764734686) [ 1494.300564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.311418] LNet: 89247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.314602] LNetError: 89247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.324788] LNet: Removed LNI 192.168.204.9@tcp [ 1494.335068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.365623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.384290] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1494.769269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.781064] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1495.322756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.9@tcp [ 1495.338342] LNetError: 89518:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1495.392353] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1495.397189] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1496.476444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.9@tcp [ 1501.328869] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:04:53 (1764734693) [ 1502.437056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg409-server [ 1503.521906] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:04:56 (1764734696) [ 1503.579861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.591513] LNet: 90459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.595126] LNetError: 90459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.598953] LNet: Removed LNI 192.168.204.9@tcp [ 1503.611160] LNet: Removed LNI 192.168.204.9@tcp1 [ 1503.624855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.653467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1503.665357] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1504.090746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1504.127979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1504.162927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1504.204352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1504.229530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1504.254658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1507.936310] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:05:00 (1764734700) [ 1508.010530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.022918] LNet: 91187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.026488] LNetError: 91187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.030783] LNetError: 91187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1508.042330] LNet: Removed LNI 192.168.204.9@tcp [ 1508.054337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.082211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.101429] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1511.843434] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:05:04 (1764734704) [ 1511.898994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.909667] LNet: 91889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.922854] LNetError: 91889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.941976] LNet: Removed LNI 192.168.204.9@tcp [ 1511.953629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.984308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.999366] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1515.534480] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:05:08 (1764734708) [ 1516.377746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1517.296755] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:05:09 (1764734709) [ 1517.521211] LNet: 92926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.524961] LNetError: 92926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.537931] LNet: Removed LNI 192.168.204.9@tcp [ 1517.824924] Key type .llcrypt unregistered [ 1517.826452] Key type ._llcrypt unregistered [ 1520.543376] Key type ._llcrypt registered [ 1520.545721] Key type .llcrypt registered [ 1520.664157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.677370] alg: No test for adler32 (adler32-zlib) [ 1521.541445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1521.561981] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1525.551497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1527.527634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.109@tcp [ 1527.713908] LNet: 93337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.718775] LNetError: 93337:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.733293] LNet: Removed LNI 192.168.204.9@tcp [ 1527.958558] Key type .llcrypt unregistered [ 1527.960400] Key type ._llcrypt unregistered [ 1533.097817] Key type ._llcrypt registered [ 1533.099381] Key type .llcrypt registered [ 1533.802746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1534.590817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1535.389642] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:05:28 (1764734728) [ 1535.564499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.572900] alg: No test for adler32 (adler32-zlib) [ 1536.410303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.452921] LNet: 94419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.799928] Key type .llcrypt unregistered [ 1536.801872] Key type ._llcrypt unregistered [ 1539.490318] Key type ._llcrypt registered [ 1539.492029] Key type .llcrypt registered [ 1539.589150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1539.597528] alg: No test for adler32 (adler32-zlib) [ 1540.440209] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1540.441780] LNet: 94671:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1540.446103] LNet: 94671:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1540.462205] LNetError: 94683:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 1540.470988] LNetError: 94683:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.13@tcp has gone from up to down [ 1540.769768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.200264] LNetError: 94683:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.10@tcp has gone from up to down [ 1559.205993] LNetError: 94683:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1559.210352] LNet: 94729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.213740] LNetError: 94729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.225699] LNet: Removed LNI 192.168.204.9@tcp [ 1559.243744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.281449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1559.298692] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1559.301872] LNet: 94739:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1559.307529] LNet: 94739:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1559.314374] LNetError: 94739:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.10@tcp is a gateway. Can not delete it [ 1559.318618] LNetError: 94739:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.204.10@tcp is a router. Can not be deleted [ 1559.328480] LNet: 94739:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1559.335450] LNetError: 94736:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 1559.352062] LNetError: 94736:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 1562.400454] LNetError: 94740:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.10@tcp has gone from up to down [ 1562.408156] LNetError: 94740:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1562.878579] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:05:55 (1764734755) [ 1563.725494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg409-server [ 1564.543145] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:05:57 (1764734757) [ 1565.389241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg409-server [ 1566.345561] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:05:58 (1764734758) [ 1567.130966] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1568.832231] Key type lgssc registered [ 1569.354964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1579.145592] Key type lgssc unregistered [ 1579.348243] LNet: 97154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.353104] LNetError: 97154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.368588] LNet: Removed LNI 192.168.204.9@tcp [ 1579.678755] Key type .llcrypt unregistered [ 1579.680489] Key type ._llcrypt unregistered [ 1586.806751] Key type ._llcrypt registered [ 1586.808872] Key type .llcrypt registered [ 1587.459841] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:06:20 (1764734780) [ 1587.613584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.621767] alg: No test for adler32 (adler32-zlib) [ 1588.456567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.502581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1588.537451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1588.568514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1588.593753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1591.614820] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:06:24 (1764734784) [ 1591.658260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.670585] LNet: 98550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.683175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.720121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.738366] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1592.177459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.187411] LNet: 98699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.190521] LNetError: 98699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.200406] LNet: Removed LNI 192.168.204.9@tcp [ 1592.209923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.240938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1592.255577] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1592.495481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.517664] LNet: Removed LNI 192.168.204.9@tcp [ 1592.529863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.558609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1592.574167] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1595.348995] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:06:28 (1764734788) [ 1595.423222] LNet: 99317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.426445] LNet: 99317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1595.430118] LNetError: 99317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.433680] LNetError: 99317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1595.443545] LNet: Removed LNI 192.168.204.9@tcp [ 1595.718488] Key type .llcrypt unregistered [ 1595.720244] Key type ._llcrypt unregistered [ 1598.087544] Key type ._llcrypt registered [ 1598.089238] Key type .llcrypt registered [ 1598.169489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1598.177348] alg: No test for adler32 (adler32-zlib) [ 1598.996765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.035589] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1599.037992] LNet: *** cfs_fail_loc=e003, val=100*** [ 1599.361151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1599.373294] Ping reply received with mapping: [ 1599.375185] found metadata reply NI[192.168.204.9@tcp] [ 1599.376927] 09 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1599.378916] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1599.380891] 5a 5a 5a 5a 5a 5a 5a 5a [ 1599.441460] LNet: 99710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.444908] LNetError: 99710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.455182] LNet: Removed LNI 192.168.204.9@tcp [ 1599.726470] Key type .llcrypt unregistered [ 1599.728129] Key type ._llcrypt unregistered [ 1604.440106] Key type ._llcrypt registered [ 1604.441580] Key type .llcrypt registered [ 1605.031488] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:06:37 (1764734797) [ 1605.281404] Key type .llcrypt unregistered [ 1605.282900] Key type ._llcrypt unregistered [ 1607.768615] Key type ._llcrypt registered [ 1607.770486] Key type .llcrypt registered [ 1607.863963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1607.871808] alg: No test for adler32 (adler32-zlib) [ 1608.771707] LNetError: 100743:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1608.955497] Key type .llcrypt unregistered [ 1608.957230] Key type ._llcrypt unregistered [ 1611.185252] Key type ._llcrypt registered [ 1611.186788] Key type .llcrypt registered [ 1611.809760] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:06:44 (1764734804) [ 1612.041368] Key type .llcrypt unregistered [ 1612.042706] Key type ._llcrypt unregistered [ 1614.402625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1614.405052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1616.647892] Key type ._llcrypt registered [ 1616.649421] Key type .llcrypt registered [ 1616.743955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1616.752428] alg: No test for adler32 (adler32-zlib) [ 1617.578672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.609106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1622.307569] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:06:55 (1764734815) [ 1622.371228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.384197] LNet: 102514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.388540] LNetError: 102514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1622.397553] LNet: Removed LNI 10.1.2.3@tcp [ 1622.409496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.753574] Key type .llcrypt unregistered [ 1622.756208] Key type ._llcrypt unregistered [ 1625.372699] Key type ._llcrypt registered [ 1625.374178] Key type .llcrypt registered [ 1625.470750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.480984] alg: No test for adler32 (adler32-zlib) [ 1626.305969] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1626.332858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1626.401099] LNet: 102799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.404231] LNetError: 102799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.417853] LNet: Removed LNI 192.168.204.9@tcp [ 1626.684404] Key type .llcrypt unregistered [ 1626.685968] Key type ._llcrypt unregistered [ 1631.185570] Key type ._llcrypt registered [ 1631.187101] Key type .llcrypt registered [ 1631.778734] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:07:04 (1764734824) [ 1632.023302] Key type .llcrypt unregistered [ 1632.025550] Key type ._llcrypt unregistered [ 1634.440643] Key type ._llcrypt registered [ 1634.442102] Key type .llcrypt registered [ 1634.528204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1634.535816] alg: No test for adler32 (adler32-zlib) [ 1637.989680] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:07:10 (1764734830) [ 1638.036146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.053066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.083858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.096787] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1638.404512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.421402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1638.439396] LNet: Removed LNI 192.168.204.9@tcp [ 1638.536774] LNet: 104532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.701401] Key type .llcrypt unregistered [ 1638.703022] Key type ._llcrypt unregistered [ 1643.179108] Key type ._llcrypt registered [ 1643.180714] Key type .llcrypt registered [ 1643.774405] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:07:16 (1764734836) [ 1643.933436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.940328] alg: No test for adler32 (adler32-zlib) [ 1644.775916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.806596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.819719] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1645.165856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1645.185148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.9@tcp [ 1645.196208] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1645.198427] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1645.200374] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1645, 0, 0 [ 1645.203468] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1645.205833] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1645.208174] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1645.210198] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1645, 0, 0 [ 1645.212991] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1645.214533] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1645.217016] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1645.218981] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1645, 0, 1695 [ 1645.224102] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1645.226707] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1645.229126] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1645.231277] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1645, 0, 1695 [ 1645.233873] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1647.947390] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:07:20 (1764734840) [ 1648.139625] LNet: 106066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.143796] LNetError: 106066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.153094] LNet: Removed LNI 192.168.204.9@tcp [ 1648.431064] Key type .llcrypt unregistered [ 1648.432671] Key type ._llcrypt unregistered [ 1650.852072] Key type ._llcrypt registered [ 1650.853651] Key type .llcrypt registered [ 1650.936652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1650.944278] alg: No test for adler32 (adler32-zlib) [ 1651.770253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1651.789032] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1655.280715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1658.699312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.720449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1658.734798] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1659.212618] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1659.249189] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1659.251546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1659.254364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1661.447584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1661.459058] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1666.187970] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:07:38 (1764734858) [ 1666.258132] LNet: 107562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.262357] LNetError: 107562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.268488] LNet: Removed LNI 192.168.204.9@tcp [ 1666.271830] LNet: Removed LNI 192.168.204.9@tcp1 [ 1666.283363] LNet: Removed LNI 10.1.2.3@tcp99 [ 1666.576296] Key type .llcrypt unregistered [ 1666.578914] Key type ._llcrypt unregistered [ 1669.179959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1669.183920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1671.421487] Key type ._llcrypt registered [ 1671.423549] Key type .llcrypt registered [ 1671.519892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1671.530170] alg: No test for adler32 (adler32-zlib) [ 1672.386069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.418270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1672.434522] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1672.815647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1672.829391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1673.926115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1673.954703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1673.980428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1673.992303] LNetError: 108305:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1674.009873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1674.037677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1674.234339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1674.270945] LNet: 108325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.276746] LNetError: 108325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.279778] LNet: Removed LNI 192.168.204.9@tcp [ 1674.287536] LNet: Removed LNI 10.1.2.3@tcp [ 1674.585281] Key type .llcrypt unregistered [ 1674.586954] Key type ._llcrypt unregistered [ 1679.723236] Key type ._llcrypt registered [ 1679.724818] Key type .llcrypt registered [ 1680.399931] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:07:53 (1764734873) [ 1680.632582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.646324] alg: No test for adler32 (adler32-zlib) [ 1681.520298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.557736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.580446] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1682.023120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1682.079215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg409-client@tcp [ 1685.516907] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:07:58 (1764734878) [ 1685.557091] LNet: 109844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.562180] LNetError: 109844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.571869] LNet: Removed LNI 192.168.204.9@tcp [ 1685.894664] Key type .llcrypt unregistered [ 1685.896450] Key type ._llcrypt unregistered [ 1688.671957] Key type ._llcrypt registered [ 1688.673710] Key type .llcrypt registered [ 1688.751385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1688.762771] alg: No test for adler32 (adler32-zlib) [ 1689.612152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1689.635324] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1689.649630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1693.088418] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:08:05 (1764734885) [ 1693.944406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1694.802765] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:08:07 (1764734887) [ 1695.678276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1696.684539] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:08:09 (1764734889) [ 1696.738580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.761443] LNet: 111260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.768257] LNetError: 111260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.780125] LNet: Removed LNI 192.168.204.9@tcp [ 1696.790802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.827731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.845527] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1697.264323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1697.288626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1700.611479] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:08:13 (1764734893) [ 1700.661993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.685527] LNet: 111971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.689663] LNetError: 111971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.706068] LNet: Removed LNI 192.168.204.9@tcp [ 1700.716878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.753470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1700.772475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1701.116830] Key type .llcrypt unregistered [ 1701.118889] Key type ._llcrypt unregistered [ 1706.121187] Key type ._llcrypt registered [ 1706.122715] Key type .llcrypt registered [ 1706.833587] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:08:19 (1764734899) [ 1707.005163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1707.017752] alg: No test for adler32 (adler32-zlib) [ 1707.880629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.918742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.950930] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1708.343694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1708.373330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1711.629331] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:08:24 (1764734904) [ 1711.677043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.691492] LNet: 113521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.695953] LNetError: 113521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.722054] LNet: Removed LNI 192.168.204.9@tcp [ 1711.735865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.774937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1711.792642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1714.600786] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:08:27 (1764734907) [ 1714.668964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.681042] LNet: 114104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.699702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.735786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1714.901319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1714.917123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1714.943710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1715.112429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1715.128927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1715.166917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.187602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.219361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1715.386809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1715.404181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1715.430905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1715.597139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1715.613544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1718.382322] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:08:31 (1764734911) [ 1718.482948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.497798] LNet: 114739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.501623] LNet: 114739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1718.518350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.552295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.566684] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1718.940284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1718.951671] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1719.358301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1719.382157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1719.479766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.491523] LNetError: 115018:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.496235] LNet: Removed LNI 192.168.204.9@tcp [ 1719.510704] LNet: Removed LNI 192.168.204.9@tcp2 [ 1719.521833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.558417] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1719.562276] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1719.565639] LNet: 115024:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.781227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1719.877393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.928204] LNet: Removed LNI 192.168.204.9@tcp [ 1720.931395] LNetError: 115061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.936239] LNetError: 115061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1720.956910] LNet: Removed LNI 192.168.204.9@tcp2 [ 1720.969777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.010397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.028532] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1721.427816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1721.440951] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1721.910956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1721.941863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1721.976537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1722.080799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.094184] LNet: 115345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.098289] LNet: 115345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1723.108702] LNet: Removed LNI 192.168.204.9@tcp [ 1723.110798] LNetError: 115345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.115192] LNetError: 115345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1723.125948] LNet: Removed LNI 192.168.204.9@tcp2 [ 1723.139636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.187698] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1723.193350] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1723.196348] LNet: 115351:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.466576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1723.572592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.583703] LNet: Removed LNI 192.168.204.9@tcp [ 1724.647964] LNet: Removed LNI 192.168.204.9@tcp2 [ 1724.658713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.687221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.702479] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1725.066548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1725.080576] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1725.566542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1725.597922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1725.716651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.752615] LNet: Removed LNI 192.168.204.9@tcp [ 1726.761598] LNet: Removed LNI 192.168.204.9@tcp2 [ 1726.774992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.820595] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1726.825879] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1726.829487] LNet: 115674:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.082047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1727.178644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.191043] LNet: 115711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.195874] LNet: 115711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1727.201465] LNetError: 115711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.205441] LNetError: 115711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1727.211121] LNet: Removed LNI 192.168.204.9@tcp [ 1727.226433] LNet: Removed LNI 192.168.204.9@tcp2 [ 1727.236941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.271201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.296741] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1727.741818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1727.765860] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1728.240484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1728.269811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1728.303971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1728.408355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.442095] LNet: Removed LNI 192.168.204.9@tcp [ 1730.472052] LNet: Removed LNI 192.168.204.9@tcp2 [ 1730.489781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.527639] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1730.531329] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1730.536155] LNet: 116001:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1730.782733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1730.875742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.885930] LNet: Removed LNI 192.168.204.9@tcp [ 1731.942786] LNet: Removed LNI 192.168.204.9@tcp2 [ 1731.960697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.002378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.019447] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1732.371545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1732.382716] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1732.894476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1732.915101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1732.947343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1733.153177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.176867] LNet: Removed LNI 192.168.204.9@tcp [ 1734.184774] LNet: Removed LNI 192.168.204.9@tcp2 [ 1734.202680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.256698] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1734.260642] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1734.545703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1734.719297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.776190] LNet: Removed LNI 192.168.204.9@tcp [ 1735.778353] LNetError: 116393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.781319] LNetError: 116393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1735.791972] LNet: Removed LNI 192.168.204.9@tcp2 [ 1735.813946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.848635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.864738] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1736.238046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1736.250830] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1736.670794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1736.694292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1736.711162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1736.739979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1736.899426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.911647] LNet: 116694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.915227] LNet: 116694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1737.952201] LNet: Removed LNI 192.168.204.9@tcp [ 1738.987736] LNet: Removed LNI 192.168.204.9@tcp2 [ 1738.998594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.032038] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1739.036580] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1739.040863] LNet: 116701:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1739.044947] LNet: 116701:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1739.279311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1742.590719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1743.371610] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:08:56 (1764734936) [ 1743.449259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.465376] LNet: Removed LNI 192.168.204.9@tcp [ 1743.482853] LNet: Removed LNI 192.168.204.9@tcp2 [ 1743.502580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.632914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.645451] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1744.001274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1744.012373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1744.435890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.445774] LNet: Removed LNI 192.168.204.9@tcp [ 1744.457769] LNet: Removed LNI 10.1.2.3@tcp [ 1744.468301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.509756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1744.525184] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1744.528298] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1744.745572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1744.760706] LNet: Removed LNI 192.168.204.9@tcp [ 1744.770947] LNet: Removed LNI 10.1.2.3@tcp [ 1747.531224] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:09:00 (1764734940) [ 1747.612329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.633830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.684296] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1747.735987] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1747.738324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1747.740776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1749.892938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.908094] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1750.253577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1750.265052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1751.252777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1751.272554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1751.306114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.9@tcp -m PUT -e local_timeout -r 1 [ 1751.341252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1751.375038] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1751.377232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1751.485884] LNetError: 118398:0:(lib-move.c:4947:lnet_parse()) 192.168.204.9@tcp, src 192.168.204.9@tcp: Dropping PUT (error -108 looking up sender) [ 1752.545154] LNetError: 118856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.550732] LNetError: 118856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1752.554744] LNet: Removed LNI 192.168.204.9@tcp [ 1752.560919] LNet: Removed LNI 10.1.2.3@tcp [ 1752.738330] Key type .llcrypt unregistered [ 1752.740060] Key type ._llcrypt unregistered [ 1757.430818] Key type ._llcrypt registered [ 1757.432361] Key type .llcrypt registered [ 1758.021259] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:09:10 (1764734950) [ 1758.171554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1758.179193] alg: No test for adler32 (adler32-zlib) [ 1758.994156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.175959] LNet: 119705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.440436] Key type .llcrypt unregistered [ 1759.441870] Key type ._llcrypt unregistered [ 1761.976878] Key type ._llcrypt registered [ 1761.978390] Key type .llcrypt registered [ 1762.059198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1762.068136] alg: No test for adler32 (adler32-zlib) [ 1762.895891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1762.911652] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1766.383754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1766.461971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1766.702898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 1766.804699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.821298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1766.838165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.854922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.871242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.887560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.903211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.919666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.929181] LNet: 120149:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.939150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.955138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.971591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.987945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.004416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.008242] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1767.021933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.287101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1767.392151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.408648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1767.426468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1767.445819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1767.461883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.478406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.494779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.510559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.520609] LNet: 120266:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.531117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.548762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.565726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.583060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.604144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.621140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.892768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 1767.991805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.007757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1768.024585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.040518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.058100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.073552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.089412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.105953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.124223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.143064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.159795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.177471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.196860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.215233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.496075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1768.604632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.621828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1768.639799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.657886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.673735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.690718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.709553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.727865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.739117] LNet: 120500:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.743383] LNet: 120500:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1768.752911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.772240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.790462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.808331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.824852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.841439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.984968] LNet: 120545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.989874] LNetError: 120545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.998687] LNet: Removed LNI 192.168.204.9@tcp [ 1769.185963] Key type .llcrypt unregistered [ 1769.187467] Key type ._llcrypt unregistered [ 1776.136852] Key type ._llcrypt registered [ 1776.138186] Key type .llcrypt registered [ 1776.825936] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:09:29 (1764734969) [ 1776.988583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.995639] alg: No test for adler32 (adler32-zlib) [ 1777.824196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.070740] LNet: 121429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.410505] Key type .llcrypt unregistered [ 1778.412116] Key type ._llcrypt unregistered [ 1781.033586] Key type ._llcrypt registered [ 1781.035023] Key type .llcrypt registered [ 1781.129604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1781.137321] alg: No test for adler32 (adler32-zlib) [ 1781.970050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1781.989395] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1785.735588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1789.914410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.931861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1789.947801] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1790.457618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1790.712602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 1790.817943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.835599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1790.853369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.872484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.894658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.915751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.937352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.960332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.969935] LNet: 122075:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.980445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.006570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.028356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.046209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.062897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.079811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.414397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1791.539948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.558256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1791.576683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.598373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.617938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.633417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.651579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.671609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.681813] LNet: 122192:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1791.693749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.711612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.734262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.750788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.772326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.798459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.126599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 1792.225589] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1792.242857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.263597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1792.283693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.300094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.316807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.344586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.395123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.414989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.432503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.448739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.465564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.487826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.505804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.522395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.851336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1792.969525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.987455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1793.004557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1793.019742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1793.038935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1793.054828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1793.074062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1793.096869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1793.108113] LNet: 122427:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1793.111026] LNet: 122427:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1793.121602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1793.145972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1793.165210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1793.182618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1793.205285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1793.233526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1795.977056] LNet: 122506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.980876] LNetError: 122506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.985099] LNet: Removed LNI 192.168.204.9@tcp [ 1795.995897] LNet: Removed LNI 192.168.204.9@tcp1 [ 1796.188797] Key type .llcrypt unregistered [ 1796.190437] Key type ._llcrypt unregistered [ 1803.132206] Key type ._llcrypt registered [ 1803.133824] Key type .llcrypt registered [ 1803.680752] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:09:56 (1764734996) [ 1806.629744] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:09:59 (1764734999) [ 1809.650764] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:10:02 (1764735002) [ 1812.741895] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:10:05 (1764735005) [ 1816.080159] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:10:08 (1764735008) [ 1819.313780] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:10:12 (1764735012) [ 1822.515624] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:10:15 (1764735015) [ 1825.579616] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:10:18 (1764735018) [ 1828.560673] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:10:21 (1764735021) [ 1831.444470] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:10:24 (1764735024) [ 1834.612995] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:10:27 (1764735027) [ 1837.881102] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:10:30 (1764735030) [ 1840.784922] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:10:33 (1764735033) [ 1843.561284] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:10:36 (1764735036) [ 1846.668493] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:10:39 (1764735039) [ 1849.698238] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:10:42 (1764735042) [ 1852.635820] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:10:45 (1764735045) [ 1855.751272] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:10:48 (1764735048) [ 1855.935018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.942665] alg: No test for adler32 (adler32-zlib) [ 1856.768303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.836663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.850500] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1857.434415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.449195] LNet: 133199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.454486] LNetError: 133199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.463538] LNet: Removed LNI 192.168.204.9@tcp [ 1857.472701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.501792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1857.517037] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1861.019603] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:10:53 (1764735053) [ 1864.421426] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:10:57 (1764735057) [ 1867.674600] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:11:00 (1764735060) [ 1870.962846] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:11:03 (1764735063) [ 1873.900211] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:11:06 (1764735066) [ 1877.184314] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:11:10 (1764735070) [ 1880.314436] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:11:13 (1764735073) [ 1883.609812] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:11:16 (1764735076) [ 1887.166740] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:11:19 (1764735079) [ 1890.812676] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:11:23 (1764735083) [ 1893.840512] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:11:26 (1764735086) [ 1896.680739] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:11:29 (1764735089) [ 1899.551249] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1681 sec ========== 23:11:32 (1764735092) [ 1899.664751] LNet: 140655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.667895] LNetError: 140655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.679313] LNet: Removed LNI 192.168.204.9@tcp [ 1899.984815] Key type .llcrypt unregistered [ 1899.986502] Key type ._llcrypt unregistered [ 1905.026551] Key type ._llcrypt registered [ 1905.028423] Key type .llcrypt registered [ 1905.207383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1905.215179] alg: No test for adler32 (adler32-zlib) [ 1906.096793] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1906.200229] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1907.800208] Key type lgssc registered [ 1908.347742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.979881] Lustre: Mounted lustre-client [ 1940.773679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1948.806768] LustreError: 143100:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8bb00466f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1948.819774] LustreError: 143100:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1948.839193] Lustre: Unmounted lustre-client [ 1962.622340] Key type lgssc unregistered [ 1962.751180] LNet: 143581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.756095] LNetError: 143581:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.817053] LNet: Removed LNI 192.168.204.9@tcp [ 1964.120646] Key type .llcrypt unregistered [ 1964.122357] Key type ._llcrypt unregistered