[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 517428590 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003294] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006999] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009142] LSM: Security Framework initializing [ 0.011016] Yama: becoming mindful. [ 0.011916] SELinux: Initializing. [ 0.013018] *** VALIDATE selinux *** [ 0.021180] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026140] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.032249] *** VALIDATE proc *** [ 0.033230] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042192] debug: unmapping init [mem 0xffffffffb8259000-0xffffffffb8260fff] [ 0.044171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.054233] rcu: Hierarchical SRCU implementation. [ 0.056311] smp: Bringing up secondary CPUs ... [ 0.057548] x86: Booting SMP configuration: [ 0.058021] .... node #0, CPUs: #1 #2 #3 [ 0.062423] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.280627] node 0 deferred pages initialised in 211ms [ 0.284086] devtmpfs: initialized [ 0.285384] x86/mm: Memory block size: 128MB [ 0.289338] gcov: version magic: 0x41383552 [ 0.291391] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.296095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.299290] pinctrl core: initialized pinctrl subsystem [ 0.302346] [ 0.303008] ************************************************************* [ 0.306022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.311018] ** ** [ 0.313015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.316013] ** ** [ 0.318013] ** This means that this kernel is built to expose internal ** [ 0.321014] ** IOMMU data structures, which may compromise security on ** [ 0.324014] ** your system. ** [ 0.326014] ** ** [ 0.329016] ** If you see this message and you are not debugging the ** [ 0.331014] ** kernel, report this immediately to your vendor! ** [ 0.334015] ** ** [ 0.337014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.340016] ************************************************************* [ 0.343661] NET: Registered protocol family 16 [ 0.345360] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.349073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.352064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.355018] cpuidle: using governor menu [ 0.356527] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.358678] PCI: Using configuration type 1 for base access [ 0.362128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.372022] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.374020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.378038] cryptd: max_cpu_qlen set to 1000 [ 0.382275] ACPI: Added _OSI(Module Device) [ 0.383013] ACPI: Added _OSI(Processor Device) [ 0.385018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.387015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.392362] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.399341] ACPI: Interpreter enabled [ 0.401140] ACPI: PM: (supports S0 S3 S4 S5) [ 0.403013] ACPI: Using IOAPIC for interrupt routing [ 0.405100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.409478] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.420466] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.422040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.424017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.427075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.432402] acpiphp: Slot [2] registered [ 0.434183] acpiphp: Slot [5] registered [ 0.435200] acpiphp: Slot [6] registered [ 0.437198] acpiphp: Slot [3] registered [ 0.439140] acpiphp: Slot [4] registered [ 0.440097] acpiphp: Slot [7] registered [ 0.441105] acpiphp: Slot [8] registered [ 0.443104] acpiphp: Slot [9] registered [ 0.444111] acpiphp: Slot [10] registered [ 0.446099] acpiphp: Slot [11] registered [ 0.447132] acpiphp: Slot [12] registered [ 0.448159] acpiphp: Slot [13] registered [ 0.449088] acpiphp: Slot [14] registered [ 0.450081] acpiphp: Slot [15] registered [ 0.451086] acpiphp: Slot [16] registered [ 0.452085] acpiphp: Slot [17] registered [ 0.454172] acpiphp: Slot [18] registered [ 0.455088] acpiphp: Slot [19] registered [ 0.456086] acpiphp: Slot [20] registered [ 0.457079] acpiphp: Slot [21] registered [ 0.458085] acpiphp: Slot [22] registered [ 0.459079] acpiphp: Slot [23] registered [ 0.460084] acpiphp: Slot [24] registered [ 0.461106] acpiphp: Slot [25] registered [ 0.463084] acpiphp: Slot [26] registered [ 0.465094] acpiphp: Slot [27] registered [ 0.466098] acpiphp: Slot [28] registered [ 0.467079] acpiphp: Slot [29] registered [ 0.468076] acpiphp: Slot [30] registered [ 0.469081] acpiphp: Slot [31] registered [ 0.470073] PCI host bridge to bus 0000:00 [ 0.471018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.473019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.475019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.476019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.478021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.480023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.481152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.483628] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.489534] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.497015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.501030] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.504020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.507026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.513460] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.515933] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.516061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.519757] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.524000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.534016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.539013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.545298] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.555017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.567017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.585017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.596427] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.605021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.611019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.625021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.638803] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.642486] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.648450] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.653278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.659139] iommu: Default domain type: Passthrough [ 0.661478] SCSI subsystem initialized [ 0.663211] ACPI: bus type USB registered [ 0.664133] usbcore: registered new interface driver usbfs [ 0.666139] usbcore: registered new interface driver hub [ 0.668099] usbcore: registered new device driver usb [ 0.672399] pps_core: LinuxPPS API ver. 1 registered [ 0.675027] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.679077] PTP clock support registered [ 0.681159] EDAC MC: Ver: 3.0.0 [ 0.684165] PCI: Using ACPI for IRQ routing [ 0.686800] NetLabel: Initializing [ 0.687014] NetLabel: domain hash size = 128 [ 0.689017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691086] NetLabel: unlabeled traffic allowed by default [ 0.694046] vgaarb: loaded [ 0.695358] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.698026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.707000] clocksource: Switched to clocksource kvm-clock [ 0.821389] VFS: Disk quotas dquot_6.6.0 [ 0.823267] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825964] *** VALIDATE ramfs *** [ 0.827490] *** VALIDATE hugetlbfs *** [ 0.829191] pnp: PnP ACPI init [ 0.831792] pnp: PnP ACPI: found 6 devices [ 0.878433] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.881833] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.884268] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.886612] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.889327] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.891157] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.893129] NET: Registered protocol family 2 [ 0.895504] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899674] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.902442] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.907171] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910496] TCP: Hash tables configured (established 65536 bind 65536) [ 0.913056] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.915202] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917786] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920518] NET: Registered protocol family 1 [ 0.923183] RPC: Registered named UNIX socket transport module. [ 0.924612] RPC: Registered udp transport module. [ 0.925811] RPC: Registered tcp transport module. [ 0.926952] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.928901] NET: Registered protocol family 44 [ 0.930321] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.931673] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933094] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.935077] PCI: CLS 0 bytes, default 64 [ 0.936474] Unpacking initramfs... [ 2.388676] debug: unmapping init [mem 0xffff91c07cc64000-0xffff91c07ffcffff] [ 2.395851] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.398294] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.402116] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.943880] Initialise system trusted keyrings [ 2.946630] Key type blacklist registered [ 2.949623] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.961112] zbud: loaded [ 2.964870] *** VALIDATE nfs *** [ 2.966309] *** VALIDATE nfs4 *** [ 2.967970] pstore: using deflate compression [ 2.972375] Platform Keyring initialized [ 3.091511] NET: Registered protocol family 38 [ 3.092818] Key type asymmetric registered [ 3.093887] Asymmetric key parser 'x509' registered [ 3.095575] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.097924] io scheduler mq-deadline registered [ 3.099885] io scheduler kyber registered [ 3.101590] io scheduler bfq registered [ 3.104311] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107575] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.110450] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.113501] ACPI: Power Button [PWRF] [ 3.120990] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.129425] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.148889] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.179897] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.211865] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.216799] Non-volatile memory driver v1.3 [ 3.218670] Linux agpgart interface v0.103 [ 3.256424] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.262142] vda: detected capacity change from 0 to 75382784 [ 3.280683] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.283425] vdb: detected capacity change from 0 to 1073741824 [ 3.293693] libphy: Fixed MDIO Bus: probed [ 3.300180] usbcore: registered new interface driver usbserial_generic [ 3.302642] usbserial: USB Serial support registered for generic [ 3.304418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.308779] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.311478] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.316618] mousedev: PS/2 mouse device common for all mice [ 3.319892] rtc_cmos 00:05: RTC can wake from S4 [ 3.323694] rtc_cmos 00:05: registered as rtc0 [ 3.325483] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.328159] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.335791] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.336372] intel_pstate: CPU model not supported [ 3.341594] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.346035] hid: raw HID events driver (C) Jiri Kosina [ 3.348788] usbcore: registered new interface driver usbhid [ 3.352657] usbhid: USB HID core driver [ 3.354503] drop_monitor: Initializing network drop monitor service [ 3.356660] Initializing XFRM netlink socket [ 3.359279] NET: Registered protocol family 10 [ 3.363175] Segment Routing with IPv6 [ 3.364450] NET: Registered protocol family 17 [ 3.366799] mpls_gso: MPLS GSO support [ 3.372347] RAS: Correctable Errors collector initialized. [ 3.374534] AVX version of gcm_enc/dec engaged. [ 3.376487] AES CTR mode by8 optimization enabled [ 3.473633] sched_clock: Marking stable (3473594168, 0)->(4418246180, -944652012) [ 3.479365] registered taskstats version 1 [ 3.482831] Loading compiled-in X.509 certificates [ 3.485217] zswap: loaded using pool lzo/zbud [ 3.519578] Key type big_key registered [ 3.534988] Key type encrypted registered [ 3.536575] ima: No TPM chip found, activating TPM-bypass! [ 3.539513] ima: Allocated hash algorithm: sha1 [ 3.541581] ima: No architecture policies found [ 3.543745] evm: Initialising EVM extended attributes: [ 3.546301] evm: security.selinux [ 3.547467] evm: security.ima [ 3.548644] evm: security.capability [ 3.550067] evm: HMAC attrs: 0x1 [ 3.553911] rtc_cmos 00:05: setting system clock to 2026-09-11 09:00:02 UTC (1789117202) [ 3.560075] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 3.563177] debug: unmapping init [mem 0xffffffffb7f82000-0xffffffffb8258fff] [ 3.571166] Write protecting the kernel read-only data: 28672k [ 3.575668] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 3.579051] debug: unmapping init [mem 0xffffffffb6f14000-0xffffffffb6ffffff] [ 3.639994] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.652272] systemd[1]: Detected virtualization kvm. [ 3.654377] systemd[1]: Detected architecture x86-64. [ 3.657216] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.683696] systemd[1]: No hostname configured. [ 3.686234] systemd[1]: Set hostname to . [ 3.688697] random: systemd: uninitialized urandom read (16 bytes read) [ 3.691679] systemd[1]: Initializing machine ID from random generator. [ 3.750291] random: ln: uninitialized urandom read (6 bytes read) [ 3.866171] random: systemd: uninitialized urandom read (16 bytes read) [ 3.870551] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.877251] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.882439] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.600843] device-mapper: uevent: version 1.0.3 [ 4.603490] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.461537] virtio_net virtio0 ens2: renamed from eth0 [ 5.482323] scsi host0: ata_piix [ 5.486422] scsi host1: ata_piix [ 5.489924] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.492268] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.063586] random: crng init done [ 10.067414] random: 7 urandom warning(s) missed due to ratelimiting [ 10.959719] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.691922] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.447334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.497238] SELinux: Disabled at runtime. [ 18.632109] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.657270] systemd[1]: Detected virtualization kvm. [ 18.659150] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.879911] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.889392] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.908462] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.918262] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.924671] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.940655] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.965237] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ 21.046560] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.510703] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.868499] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.042229] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.341618] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.937835] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.993674] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.794581] NFS: Registering the id_resolver key type [ 29.797695] Key type id_resolver registered [ 29.799568] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-client login: [ 102.060326] libcfs: loading out-of-tree module taints kernel. [ 102.109249] Key type ._llcrypt registered [ 102.112723] Key type .llcrypt registered [ 102.781712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 102.805437] alg: No test for adler32 (adler32-zlib) [ 103.067471] hrtimer: interrupt took 5508999 ns [ 104.155413] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 105.023872] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 106.911248] Key type lgssc registered [ 110.410376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 259.915341] Lustre: Mounted lustre-client - version 2.17.57_140_g76d5844 [ 265.842223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.871418] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing check_logdir /tmp/testlogs/ [ 284.424750] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing yml_node [ 285.670353] Lustre: lustre-OST0000-osc-ffff91c0c4d29000: disconnect after 23s idle [ 289.630909] Lustre: DEBUG MARKER: Client: 2.17.57.140 [ 292.284414] Lustre: DEBUG MARKER: MDS: 2.17.57.140 [ 295.336029] Lustre: DEBUG MARKER: OSS: 2.17.57.140 [ 297.370357] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 11 05:04:54 EDT 2026 [ 314.747109] Lustre: Unmounted lustre-client [ 351.828608] Key type lgssc unregistered [ 352.117764] LNet: 8417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.129092] LNetError: 8417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 352.151664] LNet: Removed LNI 192.168.204.29@tcp [ 352.973281] Key type .llcrypt unregistered [ 352.978192] Key type ._llcrypt unregistered [ 386.075075] Key type ._llcrypt registered [ 386.076919] Key type .llcrypt registered [ 386.388358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 386.418170] alg: No test for adler32 (adler32-zlib) [ 387.372505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 387.447269] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 387.500500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 391.882426] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 392.162768] LNet: 9307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.169078] LNetError: 9307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.195192] LNet: Removed LNI 192.168.204.29@tcp [ 393.051696] Key type .llcrypt unregistered [ 393.061229] Key type ._llcrypt unregistered [ 401.580507] Key type ._llcrypt registered [ 401.587895] Key type .llcrypt registered [ 402.936670] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 05:06:40 (1789117600) [ 403.519985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 403.534854] alg: No test for adler32 (adler32-zlib) [ 404.526586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.720410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.782917] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.822958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.880742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.912724] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.210741] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 05:06:49 (1789117609) [ 412.437927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.486159] LNet: 10306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.531347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.599593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 419.756655] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 05:06:57 (1789117617) [ 419.983602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.027416] LNet: 10893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.110736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.261681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 427.863816] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 05:07:05 (1789117625) [ 427.926826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.953169] LNet: 11473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.982414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.099559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 435.149599] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 05:07:12 (1789117632) [ 435.490293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.522486] LNet: 12065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.542587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.613239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 436.342891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 443.604763] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 05:07:20 (1789117640) [ 443.746146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.777220] LNet: 12652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.802819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.933188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 444.534046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 444.914405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 452.008223] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 05:07:29 (1789117649) [ 452.159761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.212371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.350580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 460.159428] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 05:07:37 (1789117657) [ 460.625985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.656613] LNet: 13837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.664605] LNet: 13837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 460.694854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.811232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 460.852840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 461.244117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 461.282047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 461.938168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 462.003097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.729144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 462.788489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 463.484310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 463.551384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 464.192164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 464.243349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.882925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 464.962335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 465.534528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 465.562948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 466.330990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 466.391963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 467.066193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 467.106125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 467.590061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 467.625476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 468.305521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 468.410440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 468.893340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 468.932239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 476.842225] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 05:07:54 (1789117674) [ 477.064612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.120737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.314310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 477.361908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 484.691574] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 05:08:02 (1789117682) [ 484.811635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.894572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.028785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 485.058486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 492.296266] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 05:08:09 (1789117689) [ 492.419424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.472681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.556845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 492.592318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 499.054924] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 05:08:16 (1789117696) [ 499.278496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.321301] LNet: 16289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.327809] LNet: 16289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 499.350823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.424236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 499.460712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 505.892760] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 05:08:23 (1789117703) [ 506.131204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.159980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.222893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 506.271475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 513.884873] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 05:08:31 (1789117711) [ 514.026898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.086868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.174710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 514.217593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 521.114192] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 05:08:38 (1789117718) [ 521.256325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.335748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.456292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 522.144977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.604774] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 05:08:46 (1789117726) [ 528.838083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.871892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.071754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 529.315864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 537.206617] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 05:08:54 (1789117734) [ 537.371864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.456445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.601508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.828260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 544.568611] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 05:09:02 (1789117742) [ 544.667423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.717525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.401077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.780875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 552.542710] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 05:09:10 (1789117750) [ 552.716321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.751608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.802950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.837406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 559.117285] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 05:09:16 (1789117756) [ 559.247607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.321346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.409536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.503863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 566.685760] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 05:09:24 (1789117764) [ 566.951426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.002074] LNet: 21730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.016562] LNet: 21730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 567.068225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.197149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 574.943714] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 05:09:32 (1789117772) [ 575.250247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.325513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.403739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 583.260304] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 05:09:40 (1789117780) [ 583.586259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.640357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.793668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 584.256721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 590.606549] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 05:09:48 (1789117788) [ 590.769568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.825237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.929770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 590.968454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 598.676932] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 05:09:56 (1789117796) [ 598.906975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.949621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.114069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 599.166571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 605.533639] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 05:10:03 (1789117803) [ 605.686862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.765658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.832802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 605.864970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 611.856454] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 05:10:09 (1789117809) [ 611.940755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.013887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.125019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 612.176869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 619.297847] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 05:10:16 (1789117816) [ 619.543122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.606244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.674902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 619.798575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 619.835331] LNetError: 25847:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 619.891665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 619.935767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 625.180973] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 05:10:23 (1789117823) [ 625.250898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.319764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.393417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zhQKRh9mmZ [ 625.437135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zhQKRh9mmZ [ 625.480344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zhQKRh9mmZ [ 630.838450] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 05:10:28 (1789117828) [ 631.181702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.228890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.342782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 631.410531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 631.477489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 637.644913] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 05:10:35 (1789117835) [ 637.782601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.858679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.031805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 638.069709] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 639.771738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 640.829646] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 642.768070] Key type lgssc registered [ 644.344753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.226811] LNetError: 27606:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.29@tcp99 has no available nets [ 650.922572] Key type lgssc unregistered [ 651.230481] LNetError: 28719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.244777] LNet: Removed LNI 192.168.204.29@tcp [ 651.651386] Key type .llcrypt unregistered [ 651.656260] Key type ._llcrypt unregistered [ 657.818511] Key type ._llcrypt registered [ 657.820467] Key type .llcrypt registered [ 659.400527] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 05:10:56 (1789117856) [ 659.699362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.717357] alg: No test for adler32 (adler32-zlib) [ 660.563470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.676222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 663.125024] LNet: Message forwarding will be enabled in 2s [ 666.209097] LNet: Message forwarding enabled [ 666.606195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 666.624229] LNet: Message forwarding will stop in 2s to 6s [ 671.062802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 671.281459] LNet: Message forwarding disabled [ 672.918812] LNet: Message forwarding will be enabled in 2s [ 676.385434] LNet: Message forwarding enabled [ 680.774488] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 05:11:18 (1789117878) [ 680.862298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.898599] LNet: 30119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.850778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.929646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 688.292892] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 05:11:26 (1789117886) [ 688.494598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.528916] LNet: 30707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.550575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.594389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 688.622873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 688.649778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 688.675683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 688.698348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 688.725487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 688.753803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 688.787468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 688.822354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 688.855648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 688.905813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 688.931544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 688.956464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 688.979661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 689.004801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 689.036280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 689.068901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 689.109465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 689.168194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 689.190488] LNetError: 30750:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 694.372983] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 05:11:32 (1789117892) [ 694.599504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.622417] LNet: 31321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.652367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.748383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.309604] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 05:11:39 (1789117899) [ 704.445872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg429-server [ 706.284369] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 05:11:43 (1789117903) [ 706.603933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.636861] LNet: 32252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.665919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.786332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.822538] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 708.037482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[30-34]@tcp [ 708.079689] LNet: 32385:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 708.141726] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 708.166487] LNetError: 32256:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.30@tcp has gone from up to down [ 709.173642] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.33@tcp is being used as a gateway but routing feature is not turned on [ 709.190944] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 710.186864] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.30@tcp is being used as a gateway but routing feature is not turned on [ 710.212810] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 712.228847] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.31@tcp is being used as a gateway but routing feature is not turned on [ 712.235207] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 716.333333] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.32@tcp is being used as a gateway but routing feature is not turned on [ 716.385324] LNetError: 32256:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 716.974530] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 05:11:54 (1789117914) [ 718.354898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg429-server [ 720.337207] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 05:11:57 (1789117917) [ 720.451256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.489411] LNet: 33315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.498220] LNetError: 33315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.535830] LNet: Removed LNI 192.168.204.29@tcp [ 720.638151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.733962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.774251] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 721.355472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 721.378965] LNet: 33438:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 721.384856] LNet: 33438:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 721.404986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[30-35/2]@tcp [ 721.422048] LNetError: 33325:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.30@tcp has gone from up to down [ 721.431802] LNetError: 33325:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 728.865463] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 05:12:06 (1789117926) [ 729.001932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.032539] LNet: 34033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.037360] LNetError: 34033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.057169] LNet: Removed LNI 192.168.204.29@tcp [ 729.118891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.273672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 729.392884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.471308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.527746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.596755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.646290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.705352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 729.805267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.850818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.919155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.985912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.021332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.087441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 730.198367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.258615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.364477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.508221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.574189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.714990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 730.864112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.908686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.009448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.170213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.266414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.371290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 731.510757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.562259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.663102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.722875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.756893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.810246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 731.846439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.880280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.935822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 738.222061] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 05:12:15 (1789117935) [ 740.375723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg429-server [ 742.408045] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 05:12:19 (1789117939) [ 744.461793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg429-server [ 746.363784] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 05:12:23 (1789117943) [ 746.671801] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.680634] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 747.479732] Key type .llcrypt unregistered [ 747.482080] Key type ._llcrypt unregistered [ 754.577232] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.815733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.826238] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.841828] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.476228] Key type ._llcrypt registered [ 757.480725] Key type .llcrypt registered [ 757.653979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.673924] alg: No test for adler32 (adler32-zlib) [ 758.631878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.717933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.764792] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 759.415644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.440696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.445921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 761.468977] LNet: Removed LNI 10.1.2.3@tcp [ 761.561430] LNet: 36161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.566517] LNetError: 36161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.580197] LNet: Removed LNI 192.168.204.29@tcp [ 762.227420] Key type .llcrypt unregistered [ 762.228785] Key type ._llcrypt unregistered [ 774.430247] Key type ._llcrypt registered [ 774.435169] Key type .llcrypt registered [ 775.950603] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 05:12:53 (1789117973) [ 776.541522] Key type .llcrypt unregistered [ 776.544877] Key type ._llcrypt unregistered [ 783.244957] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.577947] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.586255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.593339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.300474] Key type ._llcrypt registered [ 786.301981] Key type .llcrypt registered [ 786.571632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.585070] alg: No test for adler32 (adler32-zlib) [ 787.558935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.674774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.713708] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 788.588984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.606784] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.717942] LNet: 37679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.726973] LNetError: 37679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 791.734552] LNet: Removed LNI 10.1.2.3@tcp [ 791.748849] LNet: Removed LNI 192.168.204.29@tcp [ 792.462090] Key type .llcrypt unregistered [ 792.463894] Key type ._llcrypt unregistered [ 805.424913] Key type ._llcrypt registered [ 805.431461] Key type .llcrypt registered [ 806.813526] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 05:13:24 (1789118004) [ 807.474296] Key type .llcrypt unregistered [ 807.477732] Key type ._llcrypt unregistered [ 814.799929] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.944079] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.958205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.974758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.659361] Key type ._llcrypt registered [ 817.661603] Key type .llcrypt registered [ 817.883595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.898672] alg: No test for adler32 (adler32-zlib) [ 818.957975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.073403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 819.100711] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 821.378304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 821.405738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.589990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 823.622330] LNet: Removed LNI 10.1.2.3@tcp [ 823.638291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 823.664362] LNet: Removed LNI 10.1.2.31@tcp [ 823.680642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.715727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.744239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 824.768154] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 825.774410] LNet: 39611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.778364] LNetError: 39611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 825.782809] LNet: Removed LNI 10.1.2.31@tcp [ 825.792583] LNet: Removed LNI 10.1.2.3@tcp [ 826.473818] Key type .llcrypt unregistered [ 826.478646] Key type ._llcrypt unregistered [ 838.668666] Key type ._llcrypt registered [ 838.672581] Key type .llcrypt registered [ 839.987849] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 05:13:57 (1789118037) [ 840.432363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 840.447698] alg: No test for adler32 (adler32-zlib) [ 841.324025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.397239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.420396] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 842.271165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.292303] LNet: 40582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.303571] LNetError: 40582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.325080] LNet: Removed LNI 192.168.204.29@tcp [ 842.355728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.418570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.29@tcp [ 842.446521] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 843.164178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.29@tcp [ 843.206950] LNet: Removed LNI 192.168.204.29@tcp [ 843.327832] LNet: 40717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.892425] Key type .llcrypt unregistered [ 843.896995] Key type ._llcrypt unregistered [ 854.939885] Key type ._llcrypt registered [ 854.945784] Key type .llcrypt registered [ 856.462531] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 05:14:13 (1789118053) [ 858.783807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg429-server [ 860.897116] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 05:14:18 (1789118058) [ 861.515392] Key type .llcrypt unregistered [ 861.522835] Key type ._llcrypt unregistered [ 869.764395] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 869.931346] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.943901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.957140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 872.919237] Key type ._llcrypt registered [ 872.921023] Key type .llcrypt registered [ 873.124086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.139138] alg: No test for adler32 (adler32-zlib) [ 874.176090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.241694] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 875.150807] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 878.156438] LNet: 42561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.172672] LNetError: 42561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.176035] LNet: Removed LNI 192.168.204.29@tcp1 [ 878.191687] LNet: Removed LNI 10.1.2.3@tcp2 [ 879.218643] Key type .llcrypt unregistered [ 879.220716] Key type ._llcrypt unregistered [ 894.719615] Key type ._llcrypt registered [ 894.726849] Key type .llcrypt registered [ 896.268498] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 05:14:53 (1789118093) [ 896.929562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.948523] alg: No test for adler32 (adler32-zlib) [ 897.917065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.981545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.996444] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 898.917854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.947548] LNet: 43536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.954293] LNetError: 43536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.001388] LNet: Removed LNI 192.168.204.29@tcp [ 899.038297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.117309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 899.136642] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 907.060882] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 05:15:04 (1789118104) [ 907.177558] LNet: 44120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.197135] LNetError: 44120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.255232] LNet: Removed LNI 192.168.204.29@tcp [ 907.982558] Key type .llcrypt unregistered [ 907.984308] Key type ._llcrypt unregistered [ 915.646180] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 915.898330] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 915.911778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 915.914349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.702365] Key type ._llcrypt registered [ 918.715387] Key type .llcrypt registered [ 918.939675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.966800] alg: No test for adler32 (adler32-zlib) [ 919.939343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.005899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 920.032861] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 920.797088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 920.818028] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.248195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.275992] LNet: 44877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.284053] LNetError: 44877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 923.296413] LNet: Removed LNI 10.1.2.3@tcp [ 923.335053] LNet: Removed LNI 192.168.204.29@tcp [ 923.358229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.434668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 923.452302] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 923.458061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.257709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.275551] LNet: 44904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.282239] LNetError: 44904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.287403] LNetError: 44904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.294269] LNet: Removed LNI 10.1.2.3@tcp [ 924.315377] LNet: Removed LNI 192.168.204.29@tcp [ 924.358241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.448442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 924.498499] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 924.513515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.020733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 925.041295] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 926.101588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.141275] LNet: 45066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.150415] LNetError: 45066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.161035] LNetError: 45066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 926.169663] LNet: Removed LNI 10.1.2.3@tcp [ 926.171033] LNet: Removed LNI 192.168.204.29@tcp [ 926.182514] LNet: Removed LNI 192.168.204.29@tcp2 [ 926.209792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.306499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 926.362252] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 926.373689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.379914] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 933.907126] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 05:15:31 (1789118131) [ 934.161885] LNet: 45664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.167921] LNetError: 45664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 934.176375] LNetError: 45664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 934.184533] LNet: Removed LNI 10.1.2.3@tcp [ 934.187727] LNet: Removed LNI 192.168.204.29@tcp [ 934.201911] LNet: Removed LNI 192.168.204.29@tcp2 [ 934.909725] Key type .llcrypt unregistered [ 934.910931] Key type ._llcrypt unregistered [ 941.632353] Key type ._llcrypt registered [ 941.635323] Key type .llcrypt registered [ 941.874406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.886493] alg: No test for adler32 (adler32-zlib) [ 942.887748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.977616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.018963] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 944.030193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.043881] LNet: 46063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.056566] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.088445] LNet: Removed LNI 192.168.204.29@tcp [ 944.133673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.251881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.286991] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 944.890529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.912548] LNet: 46093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.917576] LNetError: 46093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.931313] LNet: Removed LNI 192.168.204.29@tcp [ 944.977088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.052597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.096375] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 945.668782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.709810] LNet: Removed LNI 192.168.204.29@tcp [ 945.725356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.794661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.824981] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 946.332532] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.477338] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.488681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.493526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.942513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.979814] LNet: 46203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.995153] LNet: 46203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 949.007336] LNetError: 46203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.027189] LNetError: 46203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 949.044821] LNet: Removed LNI 192.168.204.29@tcp [ 949.078698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.164300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 949.203411] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 956.574544] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 05:15:54 (1789118154) [ 956.885285] LNet: 46800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.897349] LNetError: 46800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.913229] LNet: Removed LNI 192.168.204.29@tcp [ 957.568908] Key type .llcrypt unregistered [ 957.571511] Key type ._llcrypt unregistered [ 964.387455] Key type ._llcrypt registered [ 964.392260] Key type .llcrypt registered [ 964.642377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.663644] alg: No test for adler32 (adler32-zlib) [ 965.585517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.652320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.696362] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 966.592519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.613387] LNet: 47203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.620452] LNetError: 47203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.647888] LNet: Removed LNI 192.168.204.29@tcp [ 966.692513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.796167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 966.836272] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 974.619621] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 05:16:11 (1789118171) [ 974.796321] LNet: 47791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.805897] LNetError: 47791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.848376] LNet: Removed LNI 192.168.204.29@tcp [ 975.728472] Key type .llcrypt unregistered [ 975.734846] Key type ._llcrypt unregistered [ 982.626275] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.785791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.794628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.810229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 985.470637] Key type ._llcrypt registered [ 985.475916] Key type .llcrypt registered [ 985.776453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.792061] alg: No test for adler32 (adler32-zlib) [ 986.751578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.818992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.839624] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 987.644638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.664365] LNet: 48250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.670263] LNetError: 48250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.695159] LNet: Removed LNI 192.168.204.29@tcp [ 987.721805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.784357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 987.823777] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 995.126443] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 05:16:32 (1789118192) [ 995.205677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.236280] LNet: 48842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.241321] LNetError: 48842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.272177] LNet: Removed LNI 192.168.204.29@tcp [ 995.295126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.416892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1002.058656] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 05:16:39 (1789118199) [ 1002.283634] LNet: 49432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.232728] Key type .llcrypt unregistered [ 1003.240051] Key type ._llcrypt unregistered [ 1010.389725] Key type ._llcrypt registered [ 1010.393047] Key type .llcrypt registered [ 1010.667449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.701610] alg: No test for adler32 (adler32-zlib) [ 1011.682701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.785878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.813416] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1013.043517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.092819] LNet: 49834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.101462] LNetError: 49834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.117035] LNet: Removed LNI 192.168.204.29@tcp [ 1013.140692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.198893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1013.218124] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1020.016417] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 05:16:57 (1789118217) [ 1020.135844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.168144] LNet: 50423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.177308] LNetError: 50423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.206587] LNet: Removed LNI 192.168.204.29@tcp [ 1020.256195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.344440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1026.146216] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 05:17:03 (1789118223) [ 1027.469905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1029.270846] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 05:17:06 (1789118226) [ 1030.800705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1032.394962] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 05:17:10 (1789118230) [ 1033.985837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1035.813070] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 05:17:13 (1789118233) [ 1037.351108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1038.893063] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 05:17:16 (1789118236) [ 1040.813170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1042.394703] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 05:17:20 (1789118240) [ 1042.480421] LNet: 52473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.098075] Key type .llcrypt unregistered [ 1043.104300] Key type ._llcrypt unregistered [ 1049.653091] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.781927] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.841944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.851440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1052.477746] Key type ._llcrypt registered [ 1052.480799] Key type .llcrypt registered [ 1052.718709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.734645] alg: No test for adler32 (adler32-zlib) [ 1053.688496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.776681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1053.804379] LNet: Added LNI 192.168.204.29@tcp [8/512/0/180] [ 1053.811429] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1053.814239] LNet: 52806:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1053.946934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.969978] LNet: 52825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.975429] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1053.982451] LNet: Removed LNI 10.1.2.3@tcp [ 1053.984716] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.004899] LNet: Removed LNI 192.168.204.29@tcp [ 1054.025642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.118224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.161431] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1054.242396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.274639] LNet: 52849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.280926] LNetError: 52849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.297629] LNet: Removed LNI 192.168.204.29@tcp [ 1054.322876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.377314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.397436] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1054.402586] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.404425] LNet: 52857:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.509787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.535522] LNet: 52875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.546294] LNetError: 52875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.557921] LNet: Removed LNI 10.1.2.3@tcp [ 1054.559588] LNetError: 52875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.588302] LNet: Removed LNI 192.168.204.29@tcp [ 1054.614481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.691302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.709427] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1054.713725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.717170] LNet: 52884:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.882965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.907950] LNet: 52910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.914603] LNetError: 52910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.924652] LNet: Removed LNI 10.1.2.3@tcp [ 1054.929090] LNetError: 52910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.955637] LNet: Removed LNI 192.168.204.29@tcp [ 1055.002598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.098556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.124906] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 1055.130953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1055.226803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.266477] LNet: 52938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.275515] LNetError: 52938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.282325] LNet: Removed LNI 10.1.2.3@tcp [ 1055.284519] LNetError: 52938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.298163] LNet: Removed LNI 192.168.204.29@tcp [ 1055.314783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.376811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.397874] LNet: Added LNI 192.168.204.29@tcp [8/256/0/182] [ 1055.403344] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1055.407377] LNet: 52946:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1055.483694] LNet: Removed LNI 10.1.2.3@tcp [ 1055.500423] LNet: Removed LNI 192.168.204.29@tcp [ 1056.223792] Key type .llcrypt unregistered [ 1056.225439] Key type ._llcrypt unregistered [ 1068.150303] Key type ._llcrypt registered [ 1068.152653] Key type .llcrypt registered [ 1069.634568] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 05:17:46 (1789118266) [ 1071.057367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1072.569381] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 05:17:50 (1789118270) [ 1073.035954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1073.051148] alg: No test for adler32 (adler32-zlib) [ 1074.018598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.082959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1074.104898] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1074.790304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.29 [ 1074.805208] LNet: 54214:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1074.811672] LNet: Removed LNI 192.168.204.29@tcp [ 1081.070777] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 05:17:58 (1789118278) [ 1081.254668] LNet: 54785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.259317] LNetError: 54785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.273563] LNet: Removed LNI 192.168.204.29@tcp [ 1082.272484] Key type .llcrypt unregistered [ 1082.276994] Key type ._llcrypt unregistered [ 1089.221862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1089.413193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.952180] Key type ._llcrypt registered [ 1091.954816] Key type .llcrypt registered [ 1092.107703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.132959] alg: No test for adler32 (adler32-zlib) [ 1093.395439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1100.475217] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 05:18:18 (1789118298) [ 1100.692761] LNet: 55698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.708929] LNetError: 55698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.755770] LNet: Removed LNI 10.1.2.3@tcp [ 1101.443176] Key type .llcrypt unregistered [ 1101.446375] Key type ._llcrypt unregistered [ 1107.484299] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1107.713349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1107.716491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1107.720783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1110.349665] Key type ._llcrypt registered [ 1110.351573] Key type .llcrypt registered [ 1110.525596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.559243] alg: No test for adler32 (adler32-zlib) [ 1111.562407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1118.139095] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 05:18:35 (1789118315) [ 1119.710247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1121.514697] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:18:38 (1789118318) [ 1121.907417] LNet: 56892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.925497] LNetError: 56892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1121.969605] LNet: Removed LNI 10.1.2.3@tcp [ 1122.718455] Key type .llcrypt unregistered [ 1122.723914] Key type ._llcrypt unregistered [ 1129.888570] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1130.150961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1130.155420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1130.159375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1133.123361] Key type ._llcrypt registered [ 1133.125806] Key type .llcrypt registered [ 1133.331685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.361662] alg: No test for adler32 (adler32-zlib) [ 1134.337989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.412977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1144.796505] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:19:01 (1789118341) [ 1145.471698] LNet: 58126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.486589] LNetError: 58126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1145.518201] LNet: Removed LNI 10.1.2.3@tcp [ 1146.352889] Key type .llcrypt unregistered [ 1146.355746] Key type ._llcrypt unregistered [ 1152.085933] Key type ._llcrypt registered [ 1152.089336] Key type .llcrypt registered [ 1152.264728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1152.275783] alg: No test for adler32 (adler32-zlib) [ 1153.303264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1153.327889] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1162.174608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1163.059740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1163.101434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1163.137510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1163.168163] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1163.404840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.433656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.454517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.488109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.525480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.554811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.616873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.617559] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1163.684206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.711402] LNet: 58569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.738706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.766848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.793532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.831195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.870492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.940929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.009955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.791497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1164.834214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1164.878575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1164.922697] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1165.384829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.420848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.477624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.552848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.633036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.678854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.740827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.788829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.812319] LNet: 58678:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.827600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.851505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.877107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.905151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.956473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.023726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.120751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.811177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1166.878132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1166.955772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1166.994978] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1167.006821] LNet: Skipped 1 previous similar message [ 1167.332657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.359629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.388533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.446046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.497568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.569617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.646222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.721617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.748728] LNet: 58787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.795598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.864630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.905944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.951155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.003727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.063718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.098673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.030192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1169.071817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1169.110819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1169.147621] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1169.393464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.458401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.516637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.579912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.639618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.689171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.770669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.845053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.888471] LNet: 58897:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.930915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.981640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.058325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.125718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.178482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.209539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.269470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.670620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1170.716179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1170.769669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1171.044721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.069356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.117847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.182385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.221468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.272071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.316468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.371392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.435169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.522303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.588059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.626393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.658386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.694398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.728721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.393263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1172.443745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1172.506400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1172.830984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.853360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.884875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.934369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.989268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.032249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.078576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.113904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.154865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.214810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.243781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.298456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.342990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.389475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.448515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.870086] LNet: 59161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.879605] LNetError: 59161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.907903] LNet: Removed LNI 192.168.204.29@tcp [ 1174.476814] Key type .llcrypt unregistered [ 1174.481713] Key type ._llcrypt unregistered [ 1193.263504] Key type ._llcrypt registered [ 1193.266610] Key type .llcrypt registered [ 1194.815544] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:19:52 (1789118392) [ 1195.715471] Key type .llcrypt unregistered [ 1195.720593] Key type ._llcrypt unregistered [ 1202.216151] Key type ._llcrypt registered [ 1202.217885] Key type .llcrypt registered [ 1202.413817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.436936] alg: No test for adler32 (adler32-zlib) [ 1203.415802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.457978] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1210.595958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1221.679950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.727352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.757612] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1223.402166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1223.476127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 1223.519174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1223.577970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 1223.625683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 1 [ 1223.663518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 1 [ 1223.714667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 1 [ 1223.755984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 1 [ 1223.811363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1223.833779] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1223.842577] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1224.078604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.115330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.158494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.194849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.225433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.259753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.289560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.332385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.356357] LNet: 60634:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.383144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.444859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.501433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.539171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.573041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.603899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.631672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.222441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1225.268637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 1225.333257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1225.367444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 1225.401793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 1 [ 1225.452881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 1 [ 1225.495645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 1 [ 1225.536341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 1 [ 1225.604442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1225.629575] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1225.634095] LNet: Skipped 2 previous similar messages [ 1225.967857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.998491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.029686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.055383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.087411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.112961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.145474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.175326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.194522] LNet: 60755:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.212214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.242229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.270137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.302171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.329589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.361415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.393666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1226.899855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1226.931637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 1226.959592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1227.000443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 1227.041823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 1 [ 1227.076423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 1 [ 1227.119984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 1 [ 1227.170963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 1 [ 1227.211179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1227.230506] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1227.239175] LNet: Skipped 2 previous similar messages [ 1227.494990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.519371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.545477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.579300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.614697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.640452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.695854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.736692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.763422] LNet: 60876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.779063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.808489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.837939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.876036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.910405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.952294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.986786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.594616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1228.644000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 1228.692269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1228.743954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 1228.787571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 1 [ 1228.834156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 1 [ 1228.870671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 1 [ 1228.909566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 1 [ 1228.964787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1229.174269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.207053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.236624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.261151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.296377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.336963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.372774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.408460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.448437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.485055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.542758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.599916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.643910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.677993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.714265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1230.243592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1230.269679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 1230.306411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1230.331936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 1230.372215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 1 [ 1230.406276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 1 [ 1230.437630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 1 [ 1230.468243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 1 [ 1230.497608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1230.528231] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1230.533192] LNet: Skipped 5 previous similar messages [ 1230.824265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.864329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.907195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.951399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.005931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.053664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.107883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.155378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.198345] LNet: 61119:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.216557] LNet: 61119:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1231.251936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.296979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.328632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.361172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.400840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.443832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.474656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.057674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1232.125638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 1232.189353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1232.232815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 1232.272758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_error -r 1 [ 1232.302986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_error -r 1 [ 1232.341463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_error -r 1 [ 1232.379471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_error -r 1 [ 1232.418951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1232.615581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.649316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.682881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.717441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.761804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.805404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.832326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.896745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.925820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.962185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.989603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.042674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.082511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.162330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.208925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.460772] LNet: 61320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.466826] LNetError: 61320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.489990] LNet: Removed LNI 192.168.204.29@tcp [ 1239.512744] LNet: Removed LNI 192.168.204.29@tcp1 [ 1239.908614] Key type .llcrypt unregistered [ 1239.912054] Key type ._llcrypt unregistered [ 1257.326739] Key type ._llcrypt registered [ 1257.328834] Key type .llcrypt registered [ 1258.387558] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:20:56 (1789118456) [ 1259.400933] Key type .llcrypt unregistered [ 1259.408228] Key type ._llcrypt unregistered [ 1265.359520] Key type ._llcrypt registered [ 1265.362991] Key type .llcrypt registered [ 1265.586940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.606566] alg: No test for adler32 (adler32-zlib) [ 1266.575673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.634552] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1275.151897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1275.744447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1275.776521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1275.803987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1275.818176] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1276.116118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.168362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.213204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.270742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.332956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.392849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.464144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.533620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.562576] LNet: 62572:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.597729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.654139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.687720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.743450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.789717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.826534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.851736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.481407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1277.515614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1277.575423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1277.595710] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1277.815294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.846621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.885824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.932559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.008807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.061447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.122912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.154553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.170931] LNet: 62682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.191650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.274208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.332693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.400415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.473049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.543100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.585882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.283890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1279.327593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1279.368899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1279.410706] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1279.701941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.729114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.753186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.804801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.845626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.902244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.962411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.017611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.036444] LNet: 62791:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.055389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.091897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.127358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.161965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.189876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.224409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.257424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.460522] LNet: 62836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.465044] LNetError: 62836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.479931] LNet: Removed LNI 192.168.204.29@tcp [ 1280.905432] Key type .llcrypt unregistered [ 1280.907357] Key type ._llcrypt unregistered [ 1302.809283] Key type ._llcrypt registered [ 1302.812754] Key type .llcrypt registered [ 1304.490953] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:21:41 (1789118501) [ 1305.453400] Key type .llcrypt unregistered [ 1305.456406] Key type ._llcrypt unregistered [ 1312.437480] Key type ._llcrypt registered [ 1312.438840] Key type .llcrypt registered [ 1312.804492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.827964] alg: No test for adler32 (adler32-zlib) [ 1313.804670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1313.827872] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1321.539912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1330.678564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.740694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.778041] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1332.777425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1332.820163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 1332.887938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1332.973113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 1333.028831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 1 [ 1333.074727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 1 [ 1333.103634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 1 [ 1333.137969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 1 [ 1333.172818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1333.205372] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1333.211493] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 1333.446030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.471816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.497990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.518641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.545632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.576813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.615907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.655441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.675958] LNet: 64307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.693828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.736234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.774345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.800402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.831279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.871274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.900566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1333.928306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.586996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1334.660658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 1334.735157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1334.799845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 1334.907422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_error -r 1 [ 1334.975370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_error -r 1 [ 1335.028142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_error -r 1 [ 1335.082784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_error -r 1 [ 1335.159737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1335.201444] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1335.208350] LNet: Skipped 2 previous similar messages [ 1335.616827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.696580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.751887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.790410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.826821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.880665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.948574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.987915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.021448] LNet: 64430:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.056622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.096944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.146409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.185637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.240529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.283787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.337632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1336.393933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.308070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1337.376854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 1337.451129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1337.516252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 1337.589133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_timeout -r 1 [ 1337.649867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_timeout -r 1 [ 1337.705326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_timeout -r 1 [ 1337.750865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_timeout -r 1 [ 1337.801748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.129@tcp [ 1337.838298] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1338.058217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.095456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.126371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.167496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.198965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.229238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.257163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.299916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.319855] LNet: 64553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.343423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.387101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.438312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.480427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.529139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.606549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.660486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1338.720610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.679467] LNet: 64635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.685821] LNetError: 64635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.690926] LNet: Removed LNI 192.168.204.29@tcp [ 1343.700837] LNet: Removed LNI 192.168.204.29@tcp1 [ 1344.055550] Key type .llcrypt unregistered [ 1344.059853] Key type ._llcrypt unregistered [ 1360.729597] Key type ._llcrypt registered [ 1360.736409] Key type .llcrypt registered [ 1362.304379] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:22:39 (1789118559) [ 1363.906133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1365.638899] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:22:43 (1789118563) [ 1366.604353] Key type .llcrypt unregistered [ 1366.609535] Key type ._llcrypt unregistered [ 1373.115776] Key type ._llcrypt registered [ 1373.117393] Key type .llcrypt registered [ 1373.293051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1373.312937] alg: No test for adler32 (adler32-zlib) [ 1374.272838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1374.316723] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1381.614206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1382.309634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1382.346264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1382.405698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1382.454191] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1382.496431] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1382.691544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1382.724432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1382.778671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1382.804425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1382.840449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1382.866839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1382.902122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.937540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1382.969291] LNet: 66183:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1382.995883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1383.041587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1383.083324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1383.145158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.183739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1383.220759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1383.541912] LNet: 66226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.549185] LNetError: 66226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.565647] LNet: Removed LNI 192.168.204.29@tcp [ 1384.045610] Key type .llcrypt unregistered [ 1384.052520] Key type ._llcrypt unregistered [ 1402.385055] Key type ._llcrypt registered [ 1402.387495] Key type .llcrypt registered [ 1402.639752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1402.650339] alg: No test for adler32 (adler32-zlib) [ 1403.640382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1403.661792] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1413.206642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1423.155354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.199975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.218843] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1424.730609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1424.780523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 1424.819333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1424.859280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 1424.926494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e network_timeout -r 1 [ 1424.984641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e network_timeout -r 1 [ 1425.030380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e network_timeout -r 1 [ 1425.061236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e network_timeout -r 1 [ 1425.111795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1425.134751] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1425.192552] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1425.200921] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 1425.391592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1425.413622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1425.448592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1425.507508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1425.555065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1425.607418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1425.653343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1425.731649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1425.753945] LNet: 67109:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1425.772469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1425.801323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1425.830774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1425.859977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1425.886955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1425.919357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1430.334560] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1430.662063] LNet: 67187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.670865] LNetError: 67187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.683366] LNet: Removed LNI 192.168.204.29@tcp [ 1430.697810] LNet: Removed LNI 192.168.204.29@tcp1 [ 1431.184789] Key type .llcrypt unregistered [ 1431.191364] Key type ._llcrypt unregistered [ 1449.481511] Key type ._llcrypt registered [ 1449.483503] Key type .llcrypt registered [ 1450.685540] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:24:08 (1789118648) [ 1451.036395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.050113] alg: No test for adler32 (adler32-zlib) [ 1451.994991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.063895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.101295] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1452.669731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.689892] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1453.504908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1453.567711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1453.626510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1453.681168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1453.741521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1453.790381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.830135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 1453.851802] LNetError: 68302:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1454.111381] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1454.112577] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1456.161313] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1457.402471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1460.256431] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1461.053503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1461.082455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1461.137592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.158487] LNet: 68409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.163128] LNetError: 68409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.172633] LNet: Removed LNI 192.168.204.29@tcp [ 1461.182874] LNet: Removed LNI 192.168.204.29@tcp1 [ 1461.204864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.256895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1461.295308] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1462.119499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1462.137798] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1463.106315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1463.140646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.214318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1463.295524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1463.354808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1463.405357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1463.497419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1463.542333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 1463.590831] LNetError: 68682:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1464.289600] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1467.233834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1470.431688] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1470.441429] LNet: Skipped 1 previous similar message [ 1475.279668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1475.324611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1475.355906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1481.492693] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:24:39 (1789118679) [ 1481.665809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.691697] LNet: 69399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.696150] LNetError: 69399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.703603] LNetError: 69399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.712883] LNet: Removed LNI 192.168.204.29@tcp [ 1481.727754] LNet: Removed LNI 192.168.204.29@tcp1 [ 1481.745186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.804100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.823876] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1482.430624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.454558] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1483.384270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1483.446398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1483.467385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1483.491638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1483.515625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1483.547556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.572731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 1483.662810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1483.808097] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.29@tcp [ 1484.832268] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1484.845842] LNet: Skipped 1 previous similar message [ 1499.117719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1499.259448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1499.291937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1499.318244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 500 [ 1499.395936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.015702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.074818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.092558] LNet: 69891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.097697] LNetError: 69891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.106397] LNetError: 69891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1506.118110] LNet: Removed LNI 192.168.204.29@tcp [ 1506.140527] LNet: Removed LNI 192.168.204.29@tcp1 [ 1506.159053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.234184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.276963] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1507.121840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.140367] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1508.032186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1508.098567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1508.188651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1508.219149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1508.247480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1508.285209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1508.326297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1508.360660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.29@tcp --health 0 [ 1509.283387] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1509.296117] LNet: Skipped 3 previous similar messages [ 1509.702972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.738098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.763848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.795638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.622597] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:25:23 (1789118723) [ 1526.124851] LNet: 70938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.130624] LNetError: 70938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.137816] LNetError: 70938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1526.142988] LNet: Removed LNI 192.168.204.29@tcp [ 1526.154825] LNet: Removed LNI 192.168.204.29@tcp1 [ 1526.744847] Key type .llcrypt unregistered [ 1526.750529] Key type ._llcrypt unregistered [ 1533.049122] Key type ._llcrypt registered [ 1533.051429] Key type .llcrypt registered [ 1533.254728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1533.269887] alg: No test for adler32 (adler32-zlib) [ 1534.188773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1534.207368] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1542.255683] Lustre: DEBUG MARKER: Initial discovery [ 1542.534363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 1545.035626] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1545.165754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1545.227141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 1545.259388] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1545.308562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1545.365854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -l 3 [ 1545.409234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.129@tcp -r 1 -m GET -e local_error [ 1545.452932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.129@tcp -r 1 -m PUT -l 6 [ 1545.504123] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1547.029776] Lustre: DEBUG MARKER: Force oleg429-client.virtnet to discover 192.168.204.129@tcp (in background) [ 1547.135130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.129@tcp [ 1548.589609] Lustre: DEBUG MARKER: Force oleg429-server to discover 192.168.204.29@tcp [ 1550.241074] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1550.829894] Lustre: DEBUG MARKER: Wait for 71699 [ 1552.377375] Lustre: DEBUG MARKER: Finished wait on 71699 [ 1552.502073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1552.563743] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1552.568457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1552.844477] LNet: 72156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.860915] LNetError: 72156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.884049] LNet: Removed LNI 192.168.204.29@tcp [ 1553.255450] Key type .llcrypt unregistered [ 1553.257905] Key type ._llcrypt unregistered [ 1567.507555] Key type ._llcrypt registered [ 1567.509826] Key type .llcrypt registered [ 1568.545678] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:26:06 (1789118766) [ 1569.108375] Key type .llcrypt unregistered [ 1569.113049] Key type ._llcrypt unregistered [ 1574.739199] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1574.882410] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1574.888250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1574.895996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1577.524075] Key type ._llcrypt registered [ 1577.536049] Key type .llcrypt registered [ 1577.768699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.780614] alg: No test for adler32 (adler32-zlib) [ 1578.704991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.766179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.793775] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1579.417915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1579.440533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1587.261401] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:26:24 (1789118784) [ 1587.399606] LNet: 74232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.404919] LNetError: 74232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1587.409263] LNet: Removed LNI 10.1.2.3@tcp [ 1587.423453] LNet: Removed LNI 192.168.204.29@tcp [ 1588.093226] Key type .llcrypt unregistered [ 1588.099798] Key type ._llcrypt unregistered [ 1594.505445] Key type ._llcrypt registered [ 1594.510708] Key type .llcrypt registered [ 1594.697674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.715086] alg: No test for adler32 (adler32-zlib) [ 1595.663701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.738169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.773989] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1596.602140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1596.621351] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1597.469168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.29@tcp [ 1598.559348] LNetError: Unexpected error -22 connecting to 192.168.204.29@tcp at host 192.168.204.29:988 [ 1602.993904] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:26:40 (1789118800) [ 1603.075186] LNet: 75364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.083895] LNetError: 75364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1603.094751] LNet: Removed LNI 10.1.2.3@tcp [ 1603.105842] LNet: Removed LNI 192.168.204.29@tcp [ 1603.737456] Key type .llcrypt unregistered [ 1603.746358] Key type ._llcrypt unregistered [ 1609.973931] Key type ._llcrypt registered [ 1609.977389] Key type .llcrypt registered [ 1610.231220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1610.258961] alg: No test for adler32 (adler32-zlib) [ 1611.191435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.257160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.283399] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1611.900603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1611.918290] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 1612.750167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.29@tcp1 --nid 192.168.204.29@tcp2 [ 1616.348443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp2 [ 1616.378643] LNetError: 76211:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.29@tcp1 to 192.168.204.29@tcp2 - Route table may be misconfigured [ 1616.394718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp2 192.168.204.29@tcp1 [ 1621.737158] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:26:59 (1789118819) [ 1621.852611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.871325] LNet: 76779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.880189] LNetError: 76779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.889949] LNet: Removed LNI 192.168.204.29@tcp1 [ 1622.961906] LNet: Removed LNI 192.168.204.29@tcp2 [ 1623.018925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.091237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.114898] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1623.797954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1623.817421] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1624.626816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 1624.672563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.29@tcp [ 1624.705824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp1 192.168.204.29@tcp1 [ 1624.745152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp -e network_timeout [ 1624.784623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -e network_timeout [ 1624.827752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -e network_timeout [ 1624.867895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -e network_timeout [ 1624.901657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1624.939124] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1624.957189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1625.076741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1625.120952] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1625.176476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1630.057800] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:27:07 (1789118827) [ 1630.233630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.249499] LNet: 77646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.256130] LNetError: 77646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.266175] LNetError: 77646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1630.272578] LNet: Removed LNI 192.168.204.29@tcp [ 1631.346384] LNet: Removed LNI 192.168.204.29@tcp1 [ 1631.370569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.466253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1631.681795] LNet: 77687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.968592] Key type .llcrypt unregistered [ 1631.975154] Key type ._llcrypt unregistered [ 1640.741475] Key type ._llcrypt registered [ 1640.743303] Key type .llcrypt registered [ 1641.770475] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:27:19 (1789118839) [ 1642.115479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.128413] alg: No test for adler32 (adler32-zlib) [ 1643.027297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.277711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1645.755785] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:27:23 (1789118843) [ 1645.951069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.984481] LNet: 78785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.014412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.074592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.097667] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1646.745438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1646.765078] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1647.693125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1647.744558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp1 [ 1647.799534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp1 [ 1653.272133] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:27:31 (1789118851) [ 1654.795951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg429-server [ 1656.231840] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:27:34 (1789118854) [ 1657.747781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg429-server [ 1659.381875] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:27:37 (1789118857) [ 1660.911200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg429-server [ 1662.372702] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:27:40 (1789118860) [ 1664.096145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg429-server [ 1665.479422] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:27:43 (1789118863) [ 1666.969366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg429-server [ 1668.385883] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:27:46 (1789118866) [ 1670.047114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg429-server [ 1671.456765] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:27:49 (1789118869) [ 1672.873113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg429-server [ 1674.215522] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:27:52 (1789118872) [ 1676.016454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg429-server [ 1677.374325] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:27:55 (1789118875) [ 1679.219594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg429-server [ 1680.479142] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:27:58 (1789118878) [ 1680.528974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.570046] LNet: 82656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.581823] LNetError: 82656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.588481] LNet: Removed LNI 192.168.204.29@tcp [ 1681.639386] LNet: Removed LNI 192.168.204.29@tcp1 [ 1681.661278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.719681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.740544] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1682.233904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1682.313390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1682.440208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.455533] LNet: 82809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.460868] LNetError: 82809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.467663] LNetError: 82809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1682.482566] LNet: Removed LNI 192.168.204.29@tcp [ 1682.504965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.553507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1682.575781] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1683.142891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1684.433144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.452420] LNetError: 82966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.476921] LNet: Removed LNI 192.168.204.29@tcp [ 1685.498526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.568119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.599847] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1686.162425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1686.224611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1686.368215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.392793] LNet: 83113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.396773] LNet: 83113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1687.461208] LNet: Removed LNI 192.168.204.29@tcp [ 1687.492175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.543939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1687.563604] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1688.214531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1688.362585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.381051] LNetError: 83258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.385986] LNetError: 83258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1688.400539] LNet: Removed LNI 192.168.204.29@tcp [ 1688.417248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.458630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.485054] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1689.116277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1689.183835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1689.371050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.416190] LNet: Removed LNI 192.168.204.29@tcp [ 1690.464469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.568560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1690.616543] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1691.230619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1691.426933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.456389] LNet: 83551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.460913] LNet: 83551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1692.521246] LNet: Removed LNI 192.168.204.29@tcp [ 1692.554404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.631784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.656170] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1693.300939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1693.374799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1694.640890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.671475] LNetError: 83707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.683188] LNetError: 83707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1694.700459] LNet: Removed LNI 192.168.204.29@tcp [ 1694.721982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.809316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1694.855045] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1695.630887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1695.879478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.935292] LNet: Removed LNI 192.168.204.29@tcp [ 1696.968723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.007952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.028195] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1697.564300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1697.629931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1699.054683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.139687] LNet: Removed LNI 192.168.204.29@tcp [ 1700.168412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.246476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1700.272316] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1700.891216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1701.040138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.051971] LNet: 84154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.055387] LNet: 84154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1702.125435] LNet: Removed LNI 192.168.204.29@tcp [ 1702.154210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.203714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.222534] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1702.803790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1702.873043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1703.114347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.137616] LNetError: 84302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.142542] LNetError: 84302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1703.154111] LNet: Removed LNI 192.168.204.29@tcp [ 1703.183478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.266572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1703.290486] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1704.051420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1704.347520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.452182] LNet: Removed LNI 192.168.204.29@tcp [ 1705.478781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.540341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.565197] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1706.317182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1706.387468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.428343] LNet: Removed LNI 192.168.204.29@tcp [ 1706.468038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.516719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.549721] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1707.400749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1707.466488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.495877] LNet: Removed LNI 192.168.204.29@tcp [ 1707.507571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.549118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1707.615129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.660795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.722146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1707.780409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.830966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.893503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1707.981447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.017727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.067261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1708.096902] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1708.831559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.866737] LNet: Removed LNI 192.168.204.29@tcp [ 1708.892990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.952838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1715.717382] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:28:33 (1789118913) [ 1715.875115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.930166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.008742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1716.030773] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 1717.153860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.200587] LNet: 85606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.206973] LNet: 85606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1717.222964] LNet: Removed LNI 192.168.204.29@tcp231 [ 1717.241446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.314283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1717.345780] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 1717.353491] LNet: 85614:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.724382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1717.771797] LNet: Removed LNI 192.168.204.29@tcp231 [ 1717.796249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1717.826481] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 1719.050725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.085557] LNet: Removed LNI 192.168.204.29@tcp231 [ 1719.102845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.187704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1719.204809] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 1719.215616] LNet: 85766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.478889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1719.519662] LNet: Removed LNI 192.168.204.29@tcp231 [ 1719.540689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1719.566321] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/0] [ 1720.605399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.630225] LNetError: 85909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.634889] LNetError: 85909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1720.648917] LNet: Removed LNI 192.168.204.29@tcp231 [ 1720.663749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.711722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1720.742768] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 1720.749487] LNet: 85917:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1721.186842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1721.222612] LNet: Removed LNI 192.168.204.29@tcp231 [ 1721.234978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1721.262484] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/60] [ 1722.501649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.544242] LNet: Removed LNI 192.168.204.29@tcp231 [ 1722.565291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.659855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1722.693926] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 1723.180547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1723.207430] LNet: Removed LNI 192.168.204.29@tcp231 [ 1723.223253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1723.244071] LNet: Added LNI 192.168.204.29@tcp231 [8/256/0/180] [ 1729.639460] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:28:47 (1789118927) [ 1729.750242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.800442] LNet: Removed LNI 192.168.204.29@tcp231 [ 1729.822546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.901421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1729.932731] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 1730.682615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1730.721374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1730.759914] LNet: Removed LNI 192.168.204.29@tcp232 [ 1730.786543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1730.814940] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 1731.442689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1731.474655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1731.505387] LNet: Removed LNI 192.168.204.29@tcp232 [ 1731.526480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1731.554053] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 1732.110334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1732.147946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1732.181600] LNet: Removed LNI 192.168.204.29@tcp232 [ 1732.208669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1732.228518] LNet: Added LNI 192.168.204.29@tcp232 [8/256/0/180] [ 1732.873369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1732.923663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1732.972550] LNet: Removed LNI 192.168.204.29@tcp232 [ 1732.987850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1738.209340] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:28:55 (1789118935) [ 1739.200436] Key type .llcrypt unregistered [ 1739.207337] Key type ._llcrypt unregistered [ 1744.190260] Key type ._llcrypt registered [ 1744.191814] Key type .llcrypt registered [ 1744.384375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1744.413069] alg: No test for adler32 (adler32-zlib) [ 1745.297390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1745.318071] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1752.005825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1760.870715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.906048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1760.936426] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1762.153611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 1762.194208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 1762.221675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 1762.251812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 1762.287108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_interrupt -r 2 [ 1762.312120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_interrupt -r 2 [ 1762.341391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_interrupt -r 2 [ 1762.379897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_interrupt -r 2 [ 1762.443516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1762.467409] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1762.475096] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1762.728631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1762.770917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1762.820297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.875319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.940503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.977970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.025408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.052818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.067374] LNet: 88571:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.083422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1763.112761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1763.138695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1763.162406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1763.193948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1763.223301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.664752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 1763.697924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 1763.730841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 1763.760491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 1763.784895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_dropped -r 2 [ 1763.813717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_dropped -r 2 [ 1763.849976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_dropped -r 2 [ 1763.894783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_dropped -r 2 [ 1763.927829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1763.946529] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1764.108389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1764.135142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1764.157268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1764.189231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1764.220256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.245687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.276830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1764.304208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1764.320465] LNet: 88692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.337471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.367957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.394296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.427498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.465772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.503474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1764.869633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 1764.896558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 1764.936775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 1764.973417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 1765.007728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_aborted -r 2 [ 1765.046911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_aborted -r 2 [ 1765.078135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_aborted -r 2 [ 1765.120149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_aborted -r 2 [ 1765.153687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1765.172633] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1765.343348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.364537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.395763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.419274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1765.448390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1765.474214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1765.501176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.533282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1765.547732] LNet: 88812:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1765.569533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1765.598385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1765.626801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.659400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.691296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.720015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.140061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 1766.179278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 1766.211673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 1766.244207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 1766.280250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_no_route -r 2 [ 1766.310305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_no_route -r 2 [ 1766.347403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_no_route -r 2 [ 1766.381716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_no_route -r 2 [ 1766.407755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1766.620118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.653938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.685263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.720869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.746413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.768696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.804069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.841404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.867122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.893312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.923545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.960048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.997475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.026820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.478222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 1767.507974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 1767.553543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 1767.593865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 1767.630947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e local_timeout -r 2 [ 1767.686728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e local_timeout -r 2 [ 1767.739729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e local_timeout -r 2 [ 1767.786883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e local_timeout -r 2 [ 1767.830055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1767.851891] LNet: There was an unexpected network error while writing to 192.168.204.129: rc = -22 [ 1767.856112] LNet: Skipped 1 previous similar message [ 1768.048540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.085599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1768.114418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.154243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.192234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.238099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.274034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.317229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.347000] LNet: 89052:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.354581] LNet: 89052:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1768.371343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.397386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.425876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.453995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.481392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.518851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.056206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 1769.111911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 1769.158024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 1769.191047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 1769.228416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp -m GET -e remote_dropped -r 2 [ 1769.269310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.29@tcp1 -d 192.168.204.129@tcp1 -m GET -e remote_dropped -r 2 [ 1769.299541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp -m GET -e remote_dropped -r 2 [ 1769.332799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.29@tcp1 -d 192.168.204.29@tcp1 -m GET -e remote_dropped -r 2 [ 1769.377054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.29@tcp 192.168.204.129@tcp [ 1769.396903] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 1769.616732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.668140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.711439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.751826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.782742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.834604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.881345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.946001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.980786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1770.017726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1770.052713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.084965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.124934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.159501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1774.310596] LNet: 89250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.320347] LNetError: 89250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.335708] LNet: Removed LNI 192.168.204.29@tcp [ 1774.355214] LNet: Removed LNI 192.168.204.29@tcp1 [ 1774.694798] Key type .llcrypt unregistered [ 1774.696506] Key type ._llcrypt unregistered [ 1789.801526] Key type ._llcrypt registered [ 1789.804491] Key type .llcrypt registered [ 1791.200180] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:29:48 (1789118988) [ 1791.679394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.689065] alg: No test for adler32 (adler32-zlib) [ 1792.607834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.686802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.710911] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1793.539409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1793.585537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.29@tcp [ 1793.622041] LNetError: 90230:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1793.695405] LNet: 1 local NIs in recovery (showing 1): 192.168.204.29@tcp [ 1793.715956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.29@tcp [ 1793.839628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.223944] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:29:57 (1789118997) [ 1800.415778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.443978] LNet: 90820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.452441] LNetError: 90820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.470654] LNet: Removed LNI 192.168.204.29@tcp [ 1800.493302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.539713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1800.578915] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1801.446608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1801.462640] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 1802.316700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.29@tcp [ 1802.334170] LNetError: 91086:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.29@tcp to recovery [ 1802.594464] LNet: There was an unexpected network error while writing to 192.168.204.29: rc = -22 [ 1803.603907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.29@tcp [ 1811.537524] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:30:08 (1789119008) [ 1813.302347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg429-server [ 1815.185374] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:30:12 (1789119012) [ 1817.328458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg429-server [ 1819.261292] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:30:16 (1789119016) [ 1820.885855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg429-server [ 1822.684985] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:30:20 (1789119020) [ 1822.840140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.878026] LNet: 92704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.906861] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.913957] LNet: Removed LNI 192.168.204.29@tcp [ 1822.938956] LNet: Removed LNI 192.168.204.29@tcp1 [ 1822.996652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.075135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.129489] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1823.843947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1823.912061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1823.989685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1824.109811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1824.161657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1824.206982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1829.778807] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:30:27 (1789119027) [ 1830.049105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.072884] LNet: 93433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.082118] LNetError: 93433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.087946] LNetError: 93433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1830.102536] LNet: Removed LNI 192.168.204.29@tcp [ 1830.118180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.174656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.207383] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1836.707386] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:30:34 (1789119034) [ 1836.770094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.794889] LNet: 94130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.804384] LNetError: 94130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.821430] LNet: Removed LNI 192.168.204.29@tcp [ 1836.850389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.905721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.924500] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1842.630502] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:30:40 (1789119040) [ 1843.720150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1845.045249] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:30:42 (1789119042) [ 1845.447955] LNet: 95170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.453584] LNetError: 95170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.473642] LNet: Removed LNI 192.168.204.29@tcp [ 1846.277965] Key type .llcrypt unregistered [ 1846.283995] Key type ._llcrypt unregistered [ 1852.016360] Key type ._llcrypt registered [ 1852.018509] Key type .llcrypt registered [ 1852.187170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1852.201955] alg: No test for adler32 (adler32-zlib) [ 1853.240794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1853.306573] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1861.715445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 1865.877859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.129@tcp [ 1866.234624] LNet: 95595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.239670] LNetError: 95595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.255901] LNet: Removed LNI 192.168.204.29@tcp [ 1866.697506] Key type .llcrypt unregistered [ 1866.700566] Key type ._llcrypt unregistered [ 1877.761567] Key type ._llcrypt registered [ 1877.765096] Key type .llcrypt registered [ 1879.000503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1880.390871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1881.894368] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:31:19 (1789119079) [ 1883.199307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1884.569473] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:31:22 (1789119082) [ 1885.971692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg429-server [ 1887.294569] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:31:25 (1789119085) [ 1888.751670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg429-server [ 1890.191597] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:31:28 (1789119088) [ 1891.399673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.413858] alg: No test for adler32 (adler32-zlib) [ 1892.490968] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 1892.679383] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1894.335408] Key type lgssc registered [ 1895.243109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.879161] Key type lgssc unregistered [ 1914.147658] LNet: 98774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.156521] LNetError: 98774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.188417] LNet: Removed LNI 192.168.204.29@tcp [ 1915.027854] Key type .llcrypt unregistered [ 1915.029776] Key type ._llcrypt unregistered [ 1932.921636] Key type ._llcrypt registered [ 1932.928759] Key type .llcrypt registered [ 1934.327265] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:32:11 (1789119131) [ 1934.824698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.837956] alg: No test for adler32 (adler32-zlib) [ 1935.749148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.813858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.863134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.901284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1935.938957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1943.151091] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:32:20 (1789119140) [ 1943.332832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.353735] LNet: 100188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.373668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.424778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.461085] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1944.602937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.641910] LNet: 100334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.654478] LNetError: 100334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.676969] LNet: Removed LNI 192.168.204.29@tcp [ 1944.691742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.756911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1944.797907] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1945.366383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.385456] LNetError: 100362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.404532] LNet: Removed LNI 192.168.204.29@tcp [ 1945.437834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.499935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1945.555180] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1952.195120] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:32:29 (1789119149) [ 1952.371569] LNet: 100951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.382650] LNet: 100951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1952.390371] LNetError: 100951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.429748] LNet: Removed LNI 192.168.204.29@tcp [ 1953.189667] Key type .llcrypt unregistered [ 1953.191526] Key type ._llcrypt unregistered [ 1958.796773] Key type ._llcrypt registered [ 1958.800161] Key type .llcrypt registered [ 1958.924708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1958.942985] alg: No test for adler32 (adler32-zlib) [ 1959.856635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.907802] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 1959.910265] LNet: *** cfs_fail_loc=e003, val=100*** [ 1960.525144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 1960.556451] LNet: *** cfs_fail_loc=e003, val=100*** [ 1960.563613] LNet: Skipped 2 previous similar messages [ 1960.576429] Ping reply received with mapping: [ 1960.580744] found metadata reply NI[192.168.204.29@tcp] [ 1960.586981] 1d cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.589629] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.592188] 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.737751] LNet: 101355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.743210] LNetError: 101355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.759919] LNet: Removed LNI 192.168.204.29@tcp [ 1961.423245] Key type .llcrypt unregistered [ 1961.427554] Key type ._llcrypt unregistered [ 1971.763535] Key type ._llcrypt registered [ 1971.767074] Key type .llcrypt registered [ 1973.212933] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:32:50 (1789119170) [ 1973.974895] Key type .llcrypt unregistered [ 1973.980957] Key type ._llcrypt unregistered [ 1980.825734] Key type ._llcrypt registered [ 1980.828134] Key type .llcrypt registered [ 1980.995907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.015300] alg: No test for adler32 (adler32-zlib) [ 1982.260892] LNetError: 102413:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1982.665522] Key type .llcrypt unregistered [ 1982.670466] Key type ._llcrypt unregistered [ 1987.934737] Key type ._llcrypt registered [ 1987.937438] Key type .llcrypt registered [ 1989.438992] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:33:06 (1789119186) [ 1990.017490] Key type .llcrypt unregistered [ 1990.019722] Key type ._llcrypt unregistered [ 1996.699972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1996.703213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1999.257666] Key type ._llcrypt registered [ 1999.259711] Key type .llcrypt registered [ 1999.408787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.440052] alg: No test for adler32 (adler32-zlib) [ 2000.392793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.456366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2009.951452] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:33:27 (1789119207) [ 2010.259150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.292761] LNet: 104202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.303822] LNetError: 104202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2010.324306] LNet: Removed LNI 10.1.2.3@tcp [ 2010.354747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.117502] Key type .llcrypt unregistered [ 2011.125069] Key type ._llcrypt unregistered [ 2018.183444] Key type ._llcrypt registered [ 2018.191707] Key type .llcrypt registered [ 2018.363969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2018.379575] alg: No test for adler32 (adler32-zlib) [ 2019.478893] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2019.555599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2019.780845] LNet: 104494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.788213] LNetError: 104494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.846821] LNet: Removed LNI 192.168.204.29@tcp [ 2021.497445] Key type .llcrypt unregistered [ 2021.500425] Key type ._llcrypt unregistered [ 2033.411329] Key type ._llcrypt registered [ 2033.412883] Key type .llcrypt registered [ 2035.191279] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:33:52 (1789119232) [ 2036.231571] Key type .llcrypt unregistered [ 2036.236539] Key type ._llcrypt unregistered [ 2044.143472] Key type ._llcrypt registered [ 2044.148794] Key type .llcrypt registered [ 2044.454989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2044.495636] alg: No test for adler32 (adler32-zlib) [ 2052.969860] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:34:10 (1789119250) [ 2053.068195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.121987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.187206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.234301] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2054.073410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.103745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2054.123222] LNet: Removed LNI 192.168.204.29@tcp [ 2054.312874] LNet: 106250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.658591] Key type .llcrypt unregistered [ 2054.662292] Key type ._llcrypt unregistered [ 2067.876394] Key type ._llcrypt registered [ 2067.881379] Key type .llcrypt registered [ 2069.495861] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:34:26 (1789119266) [ 2069.866730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.879218] alg: No test for adler32 (adler32-zlib) [ 2070.862830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.929495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.978699] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2071.766499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2071.803630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.29@tcp [ 2071.830964] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2071.840509] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2071.845644] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2071, 0, 0 [ 2071.856013] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2071.859559] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2071.864617] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2071.872705] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2071, 0, 0 [ 2071.879941] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2071.884879] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2071.897718] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2071.905370] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2071, 0, 2121 [ 2071.914084] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2071.921488] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2071.928856] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2071.936831] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2071, 0, 2121 [ 2071.947633] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2078.956503] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:34:36 (1789119276) [ 2079.530624] LNet: 107790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.555853] LNetError: 107790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.581570] LNet: Removed LNI 192.168.204.29@tcp [ 2080.289427] Key type .llcrypt unregistered [ 2080.294975] Key type ._llcrypt unregistered [ 2087.722784] Key type ._llcrypt registered [ 2087.726685] Key type .llcrypt registered [ 2088.010644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.033645] alg: No test for adler32 (adler32-zlib) [ 2088.999847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2089.026606] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2096.550405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2107.095586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.159992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2107.184326] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2108.199256] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2108.314775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2108.320437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2108.325132] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2110.723218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2110.759628] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2120.941882] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:35:18 (1789119318) [ 2121.057196] LNet: 109296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.070718] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.078724] LNet: Removed LNI 192.168.204.29@tcp [ 2122.143162] LNet: Removed LNI 192.168.204.29@tcp1 [ 2122.145834] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2122.151232] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2122.165772] LNet: Removed LNI 10.1.2.3@tcp99 [ 2122.906735] Key type .llcrypt unregistered [ 2122.909676] Key type ._llcrypt unregistered [ 2130.278040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2130.281058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2132.941455] Key type ._llcrypt registered [ 2132.944726] Key type .llcrypt registered [ 2133.252238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2133.278531] alg: No test for adler32 (adler32-zlib) [ 2134.191389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.256760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.277059] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2134.961844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2134.998946] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2137.073561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2137.127310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2137.183209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2137.227596] LNetError: 110049:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2137.271949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2137.327412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2138.053389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2138.186153] LNet: 110069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.194657] LNetError: 110069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2138.204094] LNet: Removed LNI 10.1.2.3@tcp [ 2139.243340] LNet: Removed LNI 192.168.204.29@tcp [ 2140.139423] Key type .llcrypt unregistered [ 2140.142339] Key type ._llcrypt unregistered [ 2154.821393] Key type ._llcrypt registered [ 2154.824554] Key type .llcrypt registered [ 2156.626819] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:35:53 (1789119353) [ 2157.101361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.118521] alg: No test for adler32 (adler32-zlib) [ 2158.026747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.084468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.105360] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2158.805539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2158.918447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg429-client@tcp [ 2166.170945] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:36:03 (1789119363) [ 2166.311207] LNet: 111598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.340423] LNetError: 111598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.368539] LNet: Removed LNI 192.168.204.29@tcp [ 2166.994663] Key type .llcrypt unregistered [ 2166.997808] Key type ._llcrypt unregistered [ 2172.898662] Key type ._llcrypt registered [ 2172.903434] Key type .llcrypt registered [ 2173.145984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.158287] alg: No test for adler32 (adler32-zlib) [ 2174.032647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2174.064103] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2174.093938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2180.439318] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:36:17 (1789119377) [ 2182.040800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2183.646313] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:36:21 (1789119381) [ 2184.901354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2186.800519] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:36:24 (1789119384) [ 2186.875071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.904776] LNet: 113022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.913980] LNetError: 113022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.935989] LNet: Removed LNI 192.168.204.29@tcp [ 2186.949018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.989792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.006700] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2187.840411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2187.896454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2194.326739] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:36:32 (1789119392) [ 2194.591771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.612566] LNet: 113739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.617721] LNetError: 113739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.697889] LNet: Removed LNI 192.168.204.29@tcp [ 2195.767121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.841580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2195.916280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2196.214648] LNet: 113779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.570139] Key type .llcrypt unregistered [ 2196.573716] Key type ._llcrypt unregistered [ 2207.667426] Key type ._llcrypt registered [ 2207.670566] Key type .llcrypt registered [ 2209.228267] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:36:46 (1789119406) [ 2209.645822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.659639] alg: No test for adler32 (adler32-zlib) [ 2210.588847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.631653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.662602] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2211.455820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2211.516450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2217.693528] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:36:55 (1789119415) [ 2217.817826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.866868] LNet: 115293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.874061] LNetError: 115293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.899996] LNet: Removed LNI 192.168.204.29@tcp [ 2217.957382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.030636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2218.078508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2224.324459] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:37:01 (1789119421) [ 2224.484886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.553582] LNet: 115878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.600940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.668711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2225.121767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.160715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2225.249836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2225.831721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.890161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2225.981727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.001795] LNet: 115912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.031934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.113776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2226.610652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2226.651306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2226.729381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2227.267424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2227.317373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2233.460439] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:37:11 (1789119431) [ 2233.734385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.760980] LNet: 116512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.788484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.880365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.904530] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2234.768820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2234.795778] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2235.756688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2235.804728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2236.032327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.063101] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.071412] LNet: Removed LNI 192.168.204.29@tcp [ 2236.086179] LNet: Removed LNI 192.168.204.29@tcp2 [ 2236.136624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.215496] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2236.220548] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2236.225283] LNet: 116798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2236.855573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2237.021889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.047527] LNet: Removed LNI 192.168.204.29@tcp [ 2238.052395] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.057478] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2238.084353] LNet: Removed LNI 192.168.204.29@tcp2 [ 2238.108276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.160320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.196412] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2238.900420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2238.924285] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2239.899275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2239.952246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2240.002355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2240.200321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.216070] LNet: 117119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.220282] LNet: 117119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2240.227195] LNetError: 117119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.247368] LNet: Removed LNI 192.168.204.29@tcp [ 2242.280337] LNet: Removed LNI 192.168.204.29@tcp2 [ 2242.317305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.408302] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2242.415957] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2242.424227] LNet: 117126:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2243.070399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2243.279928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.306595] LNet: Removed LNI 192.168.204.29@tcp [ 2244.328684] LNet: Removed LNI 192.168.204.29@tcp2 [ 2244.378841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.452727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.470032] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2245.106654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2245.130158] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2245.841790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2245.953448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2246.233308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.261675] LNetError: 117442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.273286] LNetError: 117442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2246.282862] LNet: Removed LNI 192.168.204.29@tcp [ 2246.289710] LNet: Removed LNI 192.168.204.29@tcp2 [ 2246.325737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.410619] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2246.418111] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2246.428295] LNet: 117448:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.889573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2247.121181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.159370] LNet: Removed LNI 192.168.204.29@tcp [ 2248.169807] LNet: Removed LNI 192.168.204.29@tcp2 [ 2248.190255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.253419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.272236] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2248.888844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2248.905822] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2249.670322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2249.727896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2249.814795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2250.041768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.085490] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.095870] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2251.167174] LNet: Removed LNI 192.168.204.29@tcp [ 2252.201211] LNet: Removed LNI 192.168.204.29@tcp2 [ 2252.227462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.298945] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2252.308124] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2252.317224] LNet: 117776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2252.831529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2253.057178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.075508] LNet: Removed LNI 192.168.204.29@tcp [ 2254.119384] LNet: Removed LNI 192.168.204.29@tcp2 [ 2254.161933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.221447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.252165] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2254.991549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2255.010468] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2255.952321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2255.984765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2256.029389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2256.363795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.386162] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.392122] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2256.400203] LNet: Removed LNI 192.168.204.29@tcp [ 2256.416146] LNet: Removed LNI 192.168.204.29@tcp2 [ 2256.444147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.511815] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2256.516100] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2256.522853] LNet: 118116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2256.993768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2257.308554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.335928] LNet: Removed LNI 192.168.204.29@tcp [ 2258.347932] LNet: Removed LNI 192.168.204.29@tcp2 [ 2258.362457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.426365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.449123] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2259.075737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2259.095826] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2259.930483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2259.976700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2260.008459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2260.077384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2260.396846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.471200] LNet: Removed LNI 192.168.204.29@tcp [ 2262.505494] LNet: Removed LNI 192.168.204.29@tcp2 [ 2262.546488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.608892] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2262.616555] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2263.088979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2269.207251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2270.741328] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:37:48 (1789119468) [ 2270.870242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.895356] LNet: 119232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.907801] LNet: 119232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2270.921322] LNet: Removed LNI 192.168.204.29@tcp [ 2270.955694] LNet: Removed LNI 192.168.204.29@tcp2 [ 2270.975470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.254436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.280024] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2272.115506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2272.133366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.082821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.115631] LNetError: 119508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2273.124688] LNetError: 119508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2273.133120] LNet: Removed LNI 10.1.2.3@tcp [ 2273.165429] LNet: Removed LNI 192.168.204.29@tcp [ 2273.197424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.274080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2273.318584] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2273.323799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.738594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2273.771464] LNet: Removed LNI 192.168.204.29@tcp [ 2273.784277] LNet: Removed LNI 10.1.2.3@tcp [ 2279.554917] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 05:37:57 (1789119477) [ 2279.698346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.740034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.795624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.822737] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2280.519455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2280.553205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.592214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.628423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.664282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.713130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.764495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.798563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.844888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.886965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2280.927120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.29@tcp [ 2281.094364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2286.528805] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:38:04 (1789119484) [ 2286.632129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.721855] LNet: Removed LNI 192.168.204.29@tcp [ 2287.757628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.875798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2288.034334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2288.039982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2288.044670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2290.359932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.379194] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2291.150171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2291.187835] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2293.330176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2293.392884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2293.473484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.29@tcp -m PUT -e local_timeout -r 1 [ 2293.546489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2293.625597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2293.630813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2293.898650] LNetError: 120914:0:(lib-move.c:4801:lnet_parse()) 192.168.204.29@tcp, src 192.168.204.29@tcp: Dropping PUT (error -108 looking up sender) [ 2294.945066] LNet: Removed LNI 10.1.2.3@tcp [ 2295.972850] LNet: Removed LNI 192.168.204.29@tcp [ 2296.361711] Key type .llcrypt unregistered [ 2296.369828] Key type ._llcrypt unregistered [ 2308.204849] Key type ._llcrypt registered [ 2308.210364] Key type .llcrypt registered [ 2309.588717] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:38:26 (1789119506) [ 2309.998552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.018563] alg: No test for adler32 (adler32-zlib) [ 2310.996856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.426628] LNet: 122230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.086964] Key type .llcrypt unregistered [ 2312.093220] Key type ._llcrypt unregistered [ 2318.994103] Key type ._llcrypt registered [ 2318.998395] Key type .llcrypt registered [ 2319.237968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.250367] alg: No test for adler32 (adler32-zlib) [ 2320.206670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2320.232604] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2328.480729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2328.749550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2329.458744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 2329.671228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2329.711572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2329.760504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2329.824707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2329.911751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2330.005535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2330.079237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2330.182627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2330.226405] LNet: 122684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.279426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2330.347522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2330.378757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2330.440347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2330.466751] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 2330.478519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2330.521113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2331.133474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2331.421361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2331.462685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2331.500281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2331.548281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2331.593680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2331.672795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2331.729477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2331.804478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2331.834716] LNet: 122801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2331.858609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2331.890605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2331.930989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2331.999546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2332.073047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2332.158612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2332.813505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 2333.032354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2333.054986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2333.079755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2333.108307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2333.136480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2333.167485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2333.201655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2333.252449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2333.286958] LNet: 122918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.317676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2333.355376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2333.412130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2333.456202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2333.488861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2333.520753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2334.050361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2334.232711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2334.270614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2334.295988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2334.321468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2334.348109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2334.378486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2334.406364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2334.435571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2334.462684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2334.510122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2334.568537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2334.595216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2334.637742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2334.671292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2335.057344] LNet: 123085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.064113] LNetError: 123085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.084277] LNet: Removed LNI 192.168.204.29@tcp [ 2335.557731] Key type .llcrypt unregistered [ 2335.562511] Key type ._llcrypt unregistered [ 2350.580290] Key type ._llcrypt registered [ 2350.584914] Key type .llcrypt registered [ 2352.292067] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:39:09 (1789119549) [ 2352.722362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.738532] alg: No test for adler32 (adler32-zlib) [ 2353.660420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.053759] LNet: 123973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.601352] Key type .llcrypt unregistered [ 2354.603308] Key type ._llcrypt unregistered [ 2360.801519] Key type ._llcrypt registered [ 2360.803192] Key type .llcrypt registered [ 2361.081231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2361.104788] alg: No test for adler32 (adler32-zlib) [ 2362.070809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2362.101105] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2370.019256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.129@tcp [ 2380.343764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.410428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2380.439066] LNet: Added LNI 192.168.204.29@tcp1 [8/256/0/180] [ 2381.347261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2381.969847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.129@tcp [ 2382.159572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2382.196539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2382.248926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2382.292719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2382.329798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2382.366829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2382.411681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2382.458709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2382.489610] LNet: 124629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2382.533249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2382.589995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2382.624242] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.129@tcp [ 2382.640811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2382.705076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2382.735659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2382.762417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2383.500572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2383.766677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2383.794205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2383.829263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2383.859306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2383.899900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2383.940274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2383.975349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2384.004696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2384.026824] LNet: 124747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2384.057524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2384.120724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2384.172522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2384.226958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2384.284761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2384.332738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2385.122151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.129@tcp [ 2385.469107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.510117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2385.563764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.646412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.696518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.747731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.808797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.844787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.870659] LNet: 124864:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.902639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.959108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.039464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.106635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2386.153377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.208312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2387.019687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2387.401568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.442456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2387.476727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.524241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.582946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.638911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.678373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.707909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.736362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.772613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.807558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.836496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.892479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.935243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2393.069264] LNet: 125066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.075572] LNetError: 125066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.083109] LNet: Removed LNI 192.168.204.29@tcp [ 2393.100334] LNet: Removed LNI 192.168.204.29@tcp1 [ 2393.483127] Key type .llcrypt unregistered [ 2393.490286] Key type ._llcrypt unregistered [ 2410.896414] Key type ._llcrypt registered [ 2410.900201] Key type .llcrypt registered [ 2412.180768] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 05:40:09 (1789119609) [ 2412.555915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.568780] alg: No test for adler32 (adler32-zlib) [ 2413.561477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.636884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.658250] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2414.515890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2414.539816] LNet: Added LNI 192.168.204.29@tcp2 [8/256/0/180] [ 2415.337353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.29@tcp [ 2415.416556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2416.479657] LNet: Removed LNI 192.168.204.29@tcp [ 2423.621931] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 05:40:20 (1789119620) [ 2425.706801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2427.677396] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:40:25 (1789119625) [ 2435.362412] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:40:32 (1789119632) [ 2442.836177] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:40:40 (1789119640) [ 2449.836983] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:40:47 (1789119647) [ 2457.026932] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:40:54 (1789119654) [ 2464.578650] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:41:02 (1789119662) [ 2472.106553] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:41:09 (1789119669) [ 2479.522776] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:41:17 (1789119677) [ 2487.039897] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:41:24 (1789119684) [ 2493.393972] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:41:30 (1789119690) [ 2500.854234] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:41:37 (1789119697) [ 2509.750712] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:41:46 (1789119706) [ 2516.836520] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:41:54 (1789119714) [ 2522.936538] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:42:00 (1789119720) [ 2529.188698] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:42:06 (1789119726) [ 2535.781448] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:42:13 (1789119733) [ 2543.537358] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:42:21 (1789119741) [ 2550.085582] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:42:27 (1789119747) [ 2550.319724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2550.390466] LNet: 136759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2550.402404] LNetError: 136759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2550.418307] LNet: Removed LNI 192.168.204.29@tcp2 [ 2550.443797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2550.581336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2550.622238] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2551.902758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2551.942854] LNet: 136901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2551.953124] LNetError: 136901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2551.974300] LNet: Removed LNI 192.168.204.29@tcp [ 2551.994185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.048530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2552.067549] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2558.777644] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:42:36 (1789119756) [ 2565.738367] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:42:43 (1789119763) [ 2572.209773] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:42:49 (1789119769) [ 2579.366947] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:42:57 (1789119777) [ 2585.985249] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:43:03 (1789119783) [ 2592.533871] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:43:10 (1789119790) [ 2599.004751] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:43:16 (1789119796) [ 2606.715245] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:43:23 (1789119803) [ 2614.942620] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:43:31 (1789119811) [ 2622.352198] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:43:39 (1789119819) [ 2629.777660] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:43:47 (1789119827) [ 2636.789829] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:43:54 (1789119834) [ 2642.579452] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2344 sec ========== 05:44:00 (1789119840) [ 2642.913795] LNet: 144364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.929083] LNetError: 144364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.942778] LNet: Removed LNI 192.168.204.29@tcp [ 2643.784740] Key type .llcrypt unregistered [ 2643.786777] Key type ._llcrypt unregistered [ 2655.868218] Key type ._llcrypt registered [ 2655.870341] Key type .llcrypt registered [ 2656.207529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2656.221664] alg: No test for adler32 (adler32-zlib) [ 2657.380203] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 2657.643187] LNet: Added LNI 192.168.204.29@tcp [8/256/0/180] [ 2659.343217] Key type lgssc registered [ 2660.518476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.754405] Lustre: Mounted lustre-client - version 2.17.57_140_g76d5844 [ 2727.200381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2733.823368] Lustre: Unmounted lustre-client [ 2763.152196] Key type lgssc unregistered [ 2763.409142] LNet: 147326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.415753] LNetError: 147326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.433686] LNet: Removed LNI 192.168.204.29@tcp [ 2764.172269] Key type .llcrypt unregistered [ 2764.179292] Key type ._llcrypt unregistered