[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475508532 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001025] APIC: Switch to symmetric I/O mode setup [ 0.002325] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.006851] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007036] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.009189] LSM: Security Framework initializing [ 0.010059] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.021082] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029334] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031053] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033242] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034196] *** VALIDATE tmpfs *** [ 0.036306] *** VALIDATE proc *** [ 0.039311] *** VALIDATE cgroup *** [ 0.041010] *** VALIDATE cgroup2 *** [ 0.043432] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047066] Spectre V2 : User space: Vulnerable [ 0.048028] Speculative Store Bypass: Vulnerable [ 0.052394] debug: unmapping init [mem 0xffffffffa2e59000-0xffffffffa2e60fff] [ 0.056000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.057347] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.058045] ... version: 2 [ 0.059029] ... bit width: 48 [ 0.060030] ... generic registers: 4 [ 0.061030] ... value mask: 0000ffffffffffff [ 0.062030] ... max period: 00007fffffffffff [ 0.063025] ... fixed-purpose events: 3 [ 0.064034] ... event mask: 000000070000000f [ 0.065542] rcu: Hierarchical SRCU implementation. [ 0.068909] smp: Bringing up secondary CPUs ... [ 0.070264] x86: Booting SMP configuration: [ 0.071075] .... node #0, CPUs: #1 #2 #3 [ 0.077126] smp: Brought up 1 node, 4 CPUs [ 0.079033] smpboot: Max logical packages: 1 [ 0.080023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.305034] node 0 deferred pages initialised in 222ms [ 0.309130] devtmpfs: initialized [ 0.310351] x86/mm: Memory block size: 128MB [ 0.315422] gcov: version magic: 0x41383552 [ 0.318342] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.321127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.324318] pinctrl core: initialized pinctrl subsystem [ 0.326194] [ 0.326581] ************************************************************* [ 0.328011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.330015] ** ** [ 0.331014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.333019] ** ** [ 0.336017] ** This means that this kernel is built to expose internal ** [ 0.338015] ** IOMMU data structures, which may compromise security on ** [ 0.340017] ** your system. ** [ 0.342013] ** ** [ 0.344014] ** If you see this message and you are not debugging the ** [ 0.347019] ** kernel, report this immediately to your vendor! ** [ 0.349017] ** ** [ 0.351021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.354019] ************************************************************* [ 0.355659] NET: Registered protocol family 16 [ 0.357438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.360071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.362072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.367068] cpuidle: using governor menu [ 0.368950] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.371852] PCI: Using configuration type 1 for base access [ 0.374181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.384154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.385118] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.388136] cryptd: max_cpu_qlen set to 1000 [ 0.392321] ACPI: Added _OSI(Module Device) [ 0.393022] ACPI: Added _OSI(Processor Device) [ 0.395022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.397025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.402075] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.408640] ACPI: Interpreter enabled [ 0.410176] ACPI: PM: (supports S0 S3 S4 S5) [ 0.412023] ACPI: Using IOAPIC for interrupt routing [ 0.415187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.419522] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.431168] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.434076] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.437032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.440144] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.448406] acpiphp: Slot [2] registered [ 0.450166] acpiphp: Slot [5] registered [ 0.451144] acpiphp: Slot [6] registered [ 0.453142] acpiphp: Slot [3] registered [ 0.454144] acpiphp: Slot [4] registered [ 0.455119] acpiphp: Slot [7] registered [ 0.456149] acpiphp: Slot [8] registered [ 0.458127] acpiphp: Slot [9] registered [ 0.459153] acpiphp: Slot [10] registered [ 0.461122] acpiphp: Slot [11] registered [ 0.462137] acpiphp: Slot [12] registered [ 0.463107] acpiphp: Slot [13] registered [ 0.465120] acpiphp: Slot [14] registered [ 0.466178] acpiphp: Slot [15] registered [ 0.467133] acpiphp: Slot [16] registered [ 0.469162] acpiphp: Slot [17] registered [ 0.470120] acpiphp: Slot [18] registered [ 0.472232] acpiphp: Slot [19] registered [ 0.474222] acpiphp: Slot [20] registered [ 0.476409] acpiphp: Slot [21] registered [ 0.478393] acpiphp: Slot [22] registered [ 0.480246] acpiphp: Slot [23] registered [ 0.482207] acpiphp: Slot [24] registered [ 0.484169] acpiphp: Slot [25] registered [ 0.486244] acpiphp: Slot [26] registered [ 0.488198] acpiphp: Slot [27] registered [ 0.492262] acpiphp: Slot [28] registered [ 0.494220] acpiphp: Slot [29] registered [ 0.496234] acpiphp: Slot [30] registered [ 0.498300] acpiphp: Slot [31] registered [ 0.500101] PCI host bridge to bus 0000:00 [ 0.502058] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.504075] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.507080] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.513070] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.515048] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.519049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.521255] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.524157] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.527378] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.534793] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.538071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.542032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.544035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.546044] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.548729] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.553839] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.556045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.560277] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.563977] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.576023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.580019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.585765] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.594022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.600024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.620028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.632614] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.648022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.665022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.678114] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.680487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.682454] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.685461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.688343] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693056] iommu: Default domain type: Passthrough [ 0.695735] SCSI subsystem initialized [ 0.697199] ACPI: bus type USB registered [ 0.700210] usbcore: registered new interface driver usbfs [ 0.701128] usbcore: registered new interface driver hub [ 0.703221] usbcore: registered new device driver usb [ 0.706301] pps_core: LinuxPPS API ver. 1 registered [ 0.708032] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.712084] PTP clock support registered [ 0.714214] EDAC MC: Ver: 3.0.0 [ 0.717167] PCI: Using ACPI for IRQ routing [ 0.719091] NetLabel: Initializing [ 0.721019] NetLabel: domain hash size = 128 [ 0.722017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.724131] NetLabel: unlabeled traffic allowed by default [ 0.727210] vgaarb: loaded [ 0.729382] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.731030] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.738247] clocksource: Switched to clocksource kvm-clock [ 0.858388] VFS: Disk quotas dquot_6.6.0 [ 0.859977] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.862512] *** VALIDATE ramfs *** [ 0.863750] *** VALIDATE hugetlbfs *** [ 0.865260] pnp: PnP ACPI init [ 0.868488] pnp: PnP ACPI: found 6 devices [ 0.893748] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.897114] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899442] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901680] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904212] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906546] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909337] NET: Registered protocol family 2 [ 0.911858] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.916994] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921715] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927588] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931464] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934549] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938215] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941300] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944826] NET: Registered protocol family 1 [ 0.948295] RPC: Registered named UNIX socket transport module. [ 0.950767] RPC: Registered udp transport module. [ 0.952752] RPC: Registered tcp transport module. [ 0.954786] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.957422] NET: Registered protocol family 44 [ 0.959155] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961516] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.963816] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.966514] PCI: CLS 0 bytes, default 64 [ 0.968561] Unpacking initramfs... [ 2.461928] debug: unmapping init [mem 0xffff9fba7cc64000-0xffff9fba7ffcffff] [ 2.466709] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.469539] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.473113] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.300529] Initialise system trusted keyrings [ 3.302283] Key type blacklist registered [ 3.304112] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.321168] zbud: loaded [ 3.325532] *** VALIDATE nfs *** [ 3.327304] *** VALIDATE nfs4 *** [ 3.329132] pstore: using deflate compression [ 3.334294] Platform Keyring initialized [ 3.447233] NET: Registered protocol family 38 [ 3.449235] Key type asymmetric registered [ 3.451477] Asymmetric key parser 'x509' registered [ 3.454086] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.459533] io scheduler mq-deadline registered [ 3.460952] io scheduler kyber registered [ 3.462536] io scheduler bfq registered [ 3.465283] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.468329] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.471207] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.474104] ACPI: Power Button [PWRF] [ 3.481036] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.489268] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.512173] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.542024] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.580030] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.586754] Non-volatile memory driver v1.3 [ 3.588863] Linux agpgart interface v0.103 [ 3.628032] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.632722] vda: detected capacity change from 0 to 76771328 [ 3.651427] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.654272] vdb: detected capacity change from 0 to 1073741824 [ 3.662851] libphy: Fixed MDIO Bus: probed [ 3.671162] usbcore: registered new interface driver usbserial_generic [ 3.674656] usbserial: USB Serial support registered for generic [ 3.677420] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.681679] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.683804] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.686316] mousedev: PS/2 mouse device common for all mice [ 3.689608] rtc_cmos 00:05: RTC can wake from S4 [ 3.692975] rtc_cmos 00:05: registered as rtc0 [ 3.694914] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.697729] intel_pstate: CPU model not supported [ 3.699934] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.706622] hid: raw HID events driver (C) Jiri Kosina [ 3.709045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.709542] usbcore: registered new interface driver usbhid [ 3.719299] usbhid: USB HID core driver [ 3.720660] drop_monitor: Initializing network drop monitor service [ 3.723162] Initializing XFRM netlink socket [ 3.723238] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.724936] NET: Registered protocol family 10 [ 3.731637] Segment Routing with IPv6 [ 3.733398] NET: Registered protocol family 17 [ 3.735609] mpls_gso: MPLS GSO support [ 3.740689] RAS: Correctable Errors collector initialized. [ 3.742898] AVX version of gcm_enc/dec engaged. [ 3.744463] AES CTR mode by8 optimization enabled [ 3.840251] sched_clock: Marking stable (3840200775, 0)->(4800314375, -960113600) [ 3.844474] registered taskstats version 1 [ 3.847340] Loading compiled-in X.509 certificates [ 3.849768] zswap: loaded using pool lzo/zbud [ 3.875353] Key type big_key registered [ 3.889410] Key type encrypted registered [ 3.890572] ima: No TPM chip found, activating TPM-bypass! [ 3.893532] ima: Allocated hash algorithm: sha1 [ 3.895287] ima: No architecture policies found [ 3.897065] evm: Initialising EVM extended attributes: [ 3.898963] evm: security.selinux [ 3.900196] evm: security.ima [ 3.901214] evm: security.capability [ 3.902462] evm: HMAC attrs: 0x1 [ 3.904690] rtc_cmos 00:05: setting system clock to 2026-09-07 15:59:06 UTC (1788796746) [ 3.910677] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.913399] debug: unmapping init [mem 0xffffffffa2b82000-0xffffffffa2e58fff] [ 3.918130] Write protecting the kernel read-only data: 28672k [ 3.921907] debug: unmapping init [mem 0xffffffffa1203000-0xffffffffa13fffff] [ 3.924166] debug: unmapping init [mem 0xffffffffa1b14000-0xffffffffa1bfffff] [ 3.957480] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.965641] systemd[1]: Detected virtualization kvm. [ 3.967831] systemd[1]: Detected architecture x86-64. [ 3.970569] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.995722] systemd[1]: No hostname configured. [ 3.997943] systemd[1]: Set hostname to . [ 4.000369] random: systemd: uninitialized urandom read (16 bytes read) [ 4.003168] systemd[1]: Initializing machine ID from random generator. [ 4.065168] random: ln: uninitialized urandom read (6 bytes read) [ 4.170408] random: systemd: uninitialized urandom read (16 bytes read) [ 4.175385] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.182446] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.188447] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.997505] device-mapper: uevent: version 1.0.3 [ 5.000178] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.792659] virtio_net virtio0 ens2: renamed from eth0 [ 5.827712] random: fast init done [ 5.857705] scsi host0: ata_piix [ 5.869240] scsi host1: ata_piix [ 5.871595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.874215] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.184026] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.407186] random: crng init done [ 10.408254] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.921356] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.128192] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.417087] SELinux: Disabled at runtime. [ 12.479791] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.487684] systemd[1]: Detected virtualization kvm. [ 12.489436] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.079574] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.083049] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.087439] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.091941] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.094843] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.102076] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.105852] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 13.170909] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.587091] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.952336] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.985126] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.014143] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.041897] EDAC sbridge: Ver: 1.1.2 [ 15.181234] Key type dns_resolver registered [ 15.490821] NFS: Registering the id_resolver key type [ 15.492822] Key type id_resolver registered [ 15.494383] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg419-client login: [ 41.736017] hrtimer: interrupt took 8283659 ns [ 84.238334] libcfs: loading out-of-tree module taints kernel. [ 84.327544] Key type ._llcrypt registered [ 84.330554] Key type .llcrypt registered [ 84.706266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 84.720544] alg: No test for adler32 (adler32-zlib) [ 86.247797] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 87.374561] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 89.199235] Key type lgssc registered [ 91.624381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 238.330570] Lustre: Mounted lustre-client - version 2.17.58_39_gc924a9e [ 244.812422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.166212] Lustre: lustre-OST0000-osc-ffff9fbac852f000: disconnect after 23s idle [ 264.287845] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing check_logdir /tmp/testlogs/ [ 269.544852] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing yml_node [ 275.177809] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 278.414813] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 281.927219] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 283.710456] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 12:03:44 EDT 2026 [ 303.180138] Lustre: Unmounted lustre-client [ 334.152522] Key type lgssc unregistered [ 334.541226] LNet: 8437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.562713] LNetError: 8437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.592771] LNet: Removed LNI 192.168.204.19@tcp [ 336.765683] Key type .llcrypt unregistered [ 336.772275] Key type ._llcrypt unregistered [ 372.713768] Key type ._llcrypt registered [ 372.716427] Key type .llcrypt registered [ 373.004347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 373.045973] alg: No test for adler32 (adler32-zlib) [ 374.025354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 374.120886] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 374.139916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 379.388183] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 379.495854] LNet: 9311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.516793] LNetError: 9311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 379.542303] LNet: Removed LNI 192.168.204.19@tcp [ 380.458829] Key type .llcrypt unregistered [ 380.465728] Key type ._llcrypt unregistered [ 388.473133] Key type ._llcrypt registered [ 388.480815] Key type .llcrypt registered [ 389.955867] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:05:30 (1788797130) [ 390.383970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 390.417103] alg: No test for adler32 (adler32-zlib) [ 391.450311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.558960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.585243] LNet: 9727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.611313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.700625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 391.716982] LNet: 9734:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 399.860552] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:05:40 (1788797140) [ 400.198872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.229829] LNet: 10312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.252147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.336306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 407.570561] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:05:48 (1788797148) [ 408.036313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.070138] LNet: 10900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.110900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.211231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 415.845310] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:05:57 (1788797157) [ 416.135557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.177613] LNet: 11484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.233175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.306842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 424.339634] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:06:05 (1788797165) [ 424.678728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.713722] LNet: 12071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.747883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.917540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 425.709515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 433.630364] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:06:14 (1788797174) [ 433.798845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.815202] LNet: 12659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.840783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.904551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 434.322930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 435.017564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 442.656219] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:06:23 (1788797183) [ 442.738773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.815161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.968338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 450.668523] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:06:31 (1788797191) [ 451.299348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.316320] LNet: 13843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.320644] LNet: 13843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 451.337937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.386490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 451.419565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 451.844414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 451.885084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 452.570815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 452.631185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 453.304950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 453.362412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 454.037294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 454.129687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 454.851644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 454.910568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 455.507952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 455.554153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 456.211431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 456.271046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 456.921682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 456.982434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 457.584210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 457.635508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 458.271731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 458.330942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 458.773662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 458.835106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 459.525236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 459.588350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 466.850170] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:06:47 (1788797207) [ 467.265693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.347949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.433432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 467.467328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 476.262680] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:06:56 (1788797216) [ 476.414923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.527551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.695912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 476.743828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 484.335426] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:07:05 (1788797225) [ 484.728861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.760620] LNet: 15699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.775244] LNet: 15699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 484.810116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.881656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 484.921548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 492.140283] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:07:12 (1788797232) [ 492.495436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.583975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.844722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 492.900906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 501.732490] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:07:22 (1788797242) [ 501.926887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.037846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.195492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 502.268496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 510.589102] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:07:31 (1788797251) [ 510.795457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.835358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.953726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 510.992651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 518.940439] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:07:39 (1788797259) [ 519.098195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.152801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.332943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.060155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.673930] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:07:49 (1788797269) [ 528.991380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.054581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.386834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 529.753135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.789899] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:07:59 (1788797279) [ 538.901900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.980447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.180712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.508107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.697657] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:08:08 (1788797288) [ 547.791926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.854294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.647319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.977935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 556.733105] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:08:17 (1788797297) [ 556.823409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.880231] LNet: 20523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.891293] LNet: 20523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 556.930377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.047276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.097588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 564.788129] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:08:25 (1788797305) [ 564.975290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.046501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.215547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.280925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 573.156903] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:08:34 (1788797314) [ 573.394629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.456740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.562046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 581.544265] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:08:42 (1788797322) [ 581.769234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.830623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.972597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 589.850567] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:08:50 (1788797330) [ 589.937965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.995211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.099779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 590.741510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 599.745255] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:09:00 (1788797340) [ 599.896929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.957344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.100414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 600.191679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 609.727823] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:09:10 (1788797350) [ 610.022273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.153990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.354830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 610.405720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 619.678986] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:09:20 (1788797360) [ 619.783604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.857967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.999680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 620.075139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 628.837671] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:09:30 (1788797370) [ 629.124887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.211721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.298214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 629.346738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 637.086416] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:09:38 (1788797378) [ 637.428292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.486274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.576500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 637.666571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 637.698293] LNetError: 25850:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 637.767528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 637.814483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 645.647812] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:09:46 (1788797386) [ 645.883686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.981292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.110479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.05nOAlOUpy [ 646.252976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.05nOAlOUpy [ 646.344297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.05nOAlOUpy [ 653.860296] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:09:54 (1788797394) [ 654.209202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.256888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.315418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 654.372857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 654.462466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 662.245713] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:10:02 (1788797402) [ 662.335390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.421226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.519886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.598312] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 664.624960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 666.081759] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 668.111898] Key type lgssc registered [ 669.701596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 674.313893] LNetError: 27609:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.19@tcp99 has no available nets [ 677.564724] Key type lgssc unregistered [ 677.954450] LNetError: 28716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.987267] LNet: Removed LNI 192.168.204.19@tcp [ 678.613689] Key type .llcrypt unregistered [ 678.621493] Key type ._llcrypt unregistered [ 686.196906] Key type ._llcrypt registered [ 686.199685] Key type .llcrypt registered [ 687.599439] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:10:28 (1788797428) [ 688.159376] Key type .llcrypt unregistered [ 688.161715] Key type ._llcrypt unregistered [ 695.802920] Key type ._llcrypt registered [ 695.808828] Key type .llcrypt registered [ 696.095700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.124565] alg: No test for adler32 (adler32-zlib) [ 697.232231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.317302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 697.336598] LNet: 29626:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 697.342842] LNet: 29626:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 697.351431] LNet: 29626:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 697.363305] LNet: Message forwarding will be enabled in 2s [ 699.425604] LNet: Message forwarding enabled [ 699.885380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 699.911633] LNet: Message forwarding will stop in 2s to 6s [ 703.524290] LNet: Message forwarding disabled [ 704.597399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 704.617228] LNet: 29715:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 704.628448] LNet: 29715:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 704.635498] LNet: 29715:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 704.648278] LNet: Message forwarding will be enabled in 2s [ 704.856184] LNet: 29728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.678681] Key type .llcrypt unregistered [ 705.682747] Key type ._llcrypt unregistered [ 719.727272] Key type ._llcrypt registered [ 719.729644] Key type .llcrypt registered [ 721.159274] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:11:02 (1788797462) [ 721.701989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.722340] alg: No test for adler32 (adler32-zlib) [ 722.793525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.983745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 730.005492] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:11:11 (1788797471) [ 730.104566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.131316] LNet: 31120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.158487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.265030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 730.353491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 730.396944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 730.466779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 730.515783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 730.558174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 730.599138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 730.674623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 730.720395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 730.818585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 730.868859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 730.927152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 730.973731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 731.037525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 731.076257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 731.136576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 731.213975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 731.294812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 731.354834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 731.383476] LNetError: 31172:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 738.630645] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:11:19 (1788797479) [ 738.739586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.799074] LNet: 31737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.846420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.978594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.668967] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:11:27 (1788797487) [ 749.025256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg419-server [ 750.863429] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:11:32 (1788797492) [ 751.381421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.405622] LNet: 32676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.447548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.513947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.538249] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 752.664263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[20-24]@tcp [ 752.685641] LNet: 32809:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 752.760900] LNetError: 32680:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.22@tcp is being used as a gateway but routing feature is not turned on [ 752.786028] LNetError: 32680:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.22@tcp has gone from up to down [ 761.495564] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:11:42 (1788797502) [ 764.825772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg419-server [ 767.273324] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:11:47 (1788797507) [ 767.543762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.591022] LNet: 33745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.599801] LNetError: 33745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.638377] LNet: Removed LNI 192.168.204.19@tcp [ 767.668260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.737977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.792121] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 769.102809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[20-25/2]@tcp [ 769.137350] LNet: 33864:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 769.145982] LNet: 33864:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 769.186775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[20-25/2]@tcp [ 769.187460] LNetError: 33750:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.20@tcp is being used as a gateway but routing feature is not turned on [ 769.233777] LNetError: 33750:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 769.258870] LNetError: 33750:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.20@tcp has gone from up to down [ 769.265861] LNetError: 33750:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 777.053648] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:11:57 (1788797517) [ 777.154282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.212641] LNet: 34452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.222429] LNetError: 34452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.273229] LNet: Removed LNI 192.168.204.19@tcp [ 777.309878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.403629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 777.470431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.501209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.557487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 777.614575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.645990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.701817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 777.868445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.947896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.101840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 778.257526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.334973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.436932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 778.678595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.734938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.839710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 779.021145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.123211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.293103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 779.484912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.528221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.611393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 779.765810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.813448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.879954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 780.023670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.104314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.279855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.439845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.516131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.635625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 780.750570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.795044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.883680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 788.431693] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:12:09 (1788797529) [ 790.764327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg419-server [ 792.405475] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:12:13 (1788797533) [ 794.475342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg419-server [ 796.379499] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:12:17 (1788797537) [ 796.667351] LNet: 35858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.671798] LNet: 35858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 797.407310] Key type .llcrypt unregistered [ 797.412829] Key type ._llcrypt unregistered [ 803.175276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.476336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.481230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.488279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.994675] Key type ._llcrypt registered [ 806.001725] Key type .llcrypt registered [ 806.187542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.205206] alg: No test for adler32 (adler32-zlib) [ 807.139467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.183989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.216061] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 807.860906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.882379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.950860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 809.965204] LNet: Removed LNI 10.1.2.3@tcp [ 810.105967] LNet: 36568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.115705] LNetError: 36568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.135559] LNet: Removed LNI 192.168.204.19@tcp [ 810.855433] Key type .llcrypt unregistered [ 810.860924] Key type ._llcrypt unregistered [ 820.652801] Key type ._llcrypt registered [ 820.655747] Key type .llcrypt registered [ 821.786449] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:12:43 (1788797563) [ 822.339936] Key type .llcrypt unregistered [ 822.341635] Key type ._llcrypt unregistered [ 828.492344] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.680248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.684162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.688232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 831.295416] Key type ._llcrypt registered [ 831.298091] Key type .llcrypt registered [ 831.460863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.479022] alg: No test for adler32 (adler32-zlib) [ 832.333665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.399658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.421795] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 833.285696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 833.312574] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.164347] LNet: 38056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.180798] LNetError: 38056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.189751] LNet: Removed LNI 10.1.2.3@tcp [ 836.206065] LNet: Removed LNI 192.168.204.19@tcp [ 836.981568] Key type .llcrypt unregistered [ 836.984648] Key type ._llcrypt unregistered [ 851.139797] Key type ._llcrypt registered [ 851.152476] Key type .llcrypt registered [ 852.997994] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:13:13 (1788797593) [ 853.643303] Key type .llcrypt unregistered [ 853.644588] Key type ._llcrypt unregistered [ 861.662264] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.799090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.812804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.821588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.643485] Key type ._llcrypt registered [ 864.646506] Key type .llcrypt registered [ 864.843798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.861688] alg: No test for adler32 (adler32-zlib) [ 865.762024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.802543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 865.828233] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 868.313440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 868.337727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.664580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 870.705631] LNet: Removed LNI 10.1.2.3@tcp [ 870.747439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 870.779394] LNet: Removed LNI 10.1.2.31@tcp [ 870.795223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 870.821850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.823613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 871.848877] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 873.016815] LNet: 39980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.027327] LNetError: 39980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 873.033477] LNet: Removed LNI 10.1.2.31@tcp [ 873.042564] LNet: Removed LNI 10.1.2.3@tcp [ 873.835728] Key type .llcrypt unregistered [ 873.837688] Key type ._llcrypt unregistered [ 888.055542] Key type ._llcrypt registered [ 888.058291] Key type .llcrypt registered [ 889.821519] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:13:50 (1788797630) [ 890.354830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.380582] alg: No test for adler32 (adler32-zlib) [ 891.403947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.480307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.509545] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 892.598920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.631850] LNet: 40930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.639958] LNetError: 40930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.654526] LNet: Removed LNI 192.168.204.19@tcp [ 892.673120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.735434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.19@tcp [ 892.770711] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 893.697120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.19@tcp [ 893.732809] LNet: Removed LNI 192.168.204.19@tcp [ 893.820558] LNet: 41065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.473488] Key type .llcrypt unregistered [ 894.476976] Key type ._llcrypt unregistered [ 906.945618] Key type ._llcrypt registered [ 906.949395] Key type .llcrypt registered [ 908.674730] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:14:09 (1788797649) [ 910.599794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg419-server [ 912.717562] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:14:13 (1788797653) [ 913.484533] Key type .llcrypt unregistered [ 913.490233] Key type ._llcrypt unregistered [ 919.533445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.767739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.773108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.783952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.421694] Key type ._llcrypt registered [ 922.424418] Key type .llcrypt registered [ 922.638187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.658274] alg: No test for adler32 (adler32-zlib) [ 923.550406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.608974] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 924.467774] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 926.767467] LNet: 42899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.780780] LNetError: 42899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.785187] LNet: Removed LNI 192.168.204.19@tcp1 [ 926.802831] LNet: Removed LNI 10.1.2.3@tcp2 [ 927.504521] Key type .llcrypt unregistered [ 927.507937] Key type ._llcrypt unregistered [ 938.920587] Key type ._llcrypt registered [ 938.928937] Key type .llcrypt registered [ 940.572613] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:14:41 (1788797681) [ 940.892854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.930625] alg: No test for adler32 (adler32-zlib) [ 941.875837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.940807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.960575] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 942.641099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.675315] LNet: 43865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.680184] LNetError: 43865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.692801] LNet: Removed LNI 192.168.204.19@tcp [ 942.722737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.804720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 942.840679] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 949.824752] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:14:50 (1788797690) [ 950.128189] LNet: 44456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.142802] LNetError: 44456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.165848] LNet: Removed LNI 192.168.204.19@tcp [ 951.040431] Key type .llcrypt unregistered [ 951.041905] Key type ._llcrypt unregistered [ 959.707526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 960.127123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 960.138608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 960.154855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.074966] Key type ._llcrypt registered [ 963.080077] Key type .llcrypt registered [ 963.397755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 963.435357] alg: No test for adler32 (adler32-zlib) [ 964.441452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.558282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.585572] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 965.387734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 965.406893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 968.362916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.387361] LNet: 45191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.401467] LNetError: 45191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.414616] LNet: Removed LNI 10.1.2.3@tcp [ 968.436140] LNet: Removed LNI 192.168.204.19@tcp [ 968.465210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.525748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 968.578508] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 968.586847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.316882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.348534] LNet: 45218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.359565] LNetError: 45218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 969.368236] LNetError: 45218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 969.378762] LNet: Removed LNI 10.1.2.3@tcp [ 969.415339] LNet: Removed LNI 192.168.204.19@tcp [ 969.496938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.578688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 969.631726] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 969.644793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.405814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 970.436640] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 971.413568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.437324] LNet: 45380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.453848] LNetError: 45380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 971.460347] LNetError: 45380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 971.468370] LNet: Removed LNI 10.1.2.3@tcp [ 971.471457] LNet: Removed LNI 192.168.204.19@tcp [ 971.498203] LNet: Removed LNI 192.168.204.19@tcp2 [ 971.555736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.643802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 971.666666] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 971.672620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 971.678175] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 980.759863] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:15:22 (1788797722) [ 981.009894] LNet: 45977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.018201] LNetError: 45977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.031642] LNetError: 45977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 981.044842] LNet: Removed LNI 10.1.2.3@tcp [ 981.047851] LNet: Removed LNI 192.168.204.19@tcp [ 981.079750] LNet: Removed LNI 192.168.204.19@tcp2 [ 982.101882] Key type .llcrypt unregistered [ 982.104120] Key type ._llcrypt unregistered [ 988.580279] Key type ._llcrypt registered [ 988.582509] Key type .llcrypt registered [ 988.792372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.815867] alg: No test for adler32 (adler32-zlib) [ 989.753785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.800991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 989.842184] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 990.647901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.669849] LNet: 46369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.676511] LNetError: 46369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.692211] LNet: Removed LNI 192.168.204.19@tcp [ 990.751463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.810965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 990.829295] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 991.213612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.230871] LNet: 46399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.234907] LNetError: 46399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.250169] LNet: Removed LNI 192.168.204.19@tcp [ 991.266516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.320974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 991.347718] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 991.964411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.016177] LNet: Removed LNI 192.168.204.19@tcp [ 992.056692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.109437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 992.142433] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 992.764127] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 992.882815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 992.885890] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 992.888771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.324875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.364907] LNet: 46508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.370560] LNet: 46508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 995.378440] LNetError: 46508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.386821] LNetError: 46508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 995.427810] LNet: Removed LNI 192.168.204.19@tcp [ 995.481078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.568271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 995.633415] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1002.067942] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:15:43 (1788797743) [ 1002.259859] LNet: 47105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.268196] LNetError: 47105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.285188] LNet: Removed LNI 192.168.204.19@tcp [ 1002.848994] Key type .llcrypt unregistered [ 1002.854597] Key type ._llcrypt unregistered [ 1009.054404] Key type ._llcrypt registered [ 1009.056543] Key type .llcrypt registered [ 1009.242160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.265641] alg: No test for adler32 (adler32-zlib) [ 1010.220387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.291606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.321603] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1011.180898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.202380] LNet: 47500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.207223] LNetError: 47500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.223101] LNet: Removed LNI 192.168.204.19@tcp [ 1011.240322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.303994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1011.332287] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1018.019079] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:15:59 (1788797759) [ 1018.143582] LNet: 48084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.161255] LNetError: 48084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.187113] LNet: Removed LNI 192.168.204.19@tcp [ 1018.854740] Key type .llcrypt unregistered [ 1018.864246] Key type ._llcrypt unregistered [ 1024.754967] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.933280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1024.943525] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1024.961333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1027.629780] Key type ._llcrypt registered [ 1027.632712] Key type .llcrypt registered [ 1027.857508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.871081] alg: No test for adler32 (adler32-zlib) [ 1028.792571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.853240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.878659] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1029.784745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.797592] LNet: 48539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.800816] LNetError: 48539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.813784] LNet: Removed LNI 192.168.204.19@tcp [ 1029.850350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.916353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1029.955356] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1037.112709] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:16:18 (1788797778) [ 1037.455873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.483394] LNet: 49138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.498401] LNetError: 49138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.513794] LNet: Removed LNI 192.168.204.19@tcp [ 1037.544398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.665958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1043.280914] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:16:24 (1788797784) [ 1043.819993] LNet: 49721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.436545] Key type .llcrypt unregistered [ 1044.439604] Key type ._llcrypt unregistered [ 1050.233349] Key type ._llcrypt registered [ 1050.235677] Key type .llcrypt registered [ 1050.466749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.488252] alg: No test for adler32 (adler32-zlib) [ 1051.490140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.560521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.587892] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1052.412219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.451325] LNet: 50115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.458922] LNetError: 50115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.487483] LNet: Removed LNI 192.168.204.19@tcp [ 1052.521834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.600743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1052.625329] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1059.246599] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:16:40 (1788797800) [ 1059.343307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.373679] LNet: 50703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.378772] LNetError: 50703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.395593] LNet: Removed LNI 192.168.204.19@tcp [ 1059.414766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.505219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1065.254108] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:16:46 (1788797806) [ 1066.924188] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1068.853554] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:16:49 (1788797809) [ 1070.476394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1072.221768] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:16:53 (1788797813) [ 1073.607973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1075.164708] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:16:56 (1788797816) [ 1076.525086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1078.242767] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:16:59 (1788797819) [ 1080.106380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1082.273307] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:17:03 (1788797823) [ 1082.581737] LNet: 52730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.196588] Key type .llcrypt unregistered [ 1083.202665] Key type ._llcrypt unregistered [ 1090.244738] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1090.539780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1090.545915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1090.549844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1093.177833] Key type ._llcrypt registered [ 1093.180374] Key type .llcrypt registered [ 1093.333929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1093.349392] alg: No test for adler32 (adler32-zlib) [ 1094.286289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.354140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1094.371720] LNet: Added LNI 192.168.204.19@tcp [8/512/0/180] [ 1094.378978] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1094.381763] LNet: 53048:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1094.460450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1094.481515] LNet: 53067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.489212] LNetError: 53067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1094.496925] LNet: Removed LNI 10.1.2.3@tcp [ 1094.499935] LNetError: 53067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.516881] LNet: Removed LNI 192.168.204.19@tcp [ 1094.537793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.615640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1094.637165] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1094.689450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1094.707913] LNet: 53091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.715593] LNetError: 53091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.729883] LNet: Removed LNI 192.168.204.19@tcp [ 1094.749000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.821493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1094.850846] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1094.856468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.860810] LNet: 53099:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1094.942298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1094.962708] LNet: 53117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.969782] LNetError: 53117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1094.979338] LNet: Removed LNI 10.1.2.3@tcp [ 1094.981962] LNetError: 53117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.993641] LNet: Removed LNI 192.168.204.19@tcp [ 1095.011962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.062490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1095.085409] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1095.092400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1095.094820] LNet: 53125:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1095.266817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.298867] LNet: 53151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.307960] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.315518] LNet: Removed LNI 10.1.2.3@tcp [ 1095.321217] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.349556] LNet: Removed LNI 192.168.204.19@tcp [ 1095.393402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.488794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1095.517552] LNet: Added LNI 192.168.204.19@tcp [8/256/0/182] [ 1095.522574] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1095.622809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.645306] LNet: 53179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.649956] LNetError: 53179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.654710] LNet: Removed LNI 10.1.2.3@tcp [ 1095.656954] LNetError: 53179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.665886] LNet: Removed LNI 192.168.204.19@tcp [ 1095.681698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.742137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1095.773435] LNet: Added LNI 192.168.204.19@tcp [8/256/0/182] [ 1095.778799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1095.781951] LNet: 53187:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1095.839834] LNet: Removed LNI 10.1.2.3@tcp [ 1095.855277] LNet: Removed LNI 192.168.204.19@tcp [ 1096.709744] Key type .llcrypt unregistered [ 1096.714722] Key type ._llcrypt unregistered [ 1108.072232] Key type ._llcrypt registered [ 1108.074711] Key type .llcrypt registered [ 1109.535491] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:17:30 (1788797850) [ 1110.997405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1113.075717] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:17:34 (1788797854) [ 1113.503435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1113.527785] alg: No test for adler32 (adler32-zlib) [ 1114.477133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.543519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1114.561138] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1115.392518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.19 [ 1115.415530] LNet: 54448:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1115.423293] LNet: Removed LNI 192.168.204.19@tcp [ 1121.386583] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:17:42 (1788797862) [ 1121.550996] LNet: 55015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.558916] LNetError: 55015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.582491] LNet: Removed LNI 192.168.204.19@tcp [ 1122.235971] Key type .llcrypt unregistered [ 1122.240901] Key type ._llcrypt unregistered [ 1128.750222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1129.038034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1131.705115] Key type ._llcrypt registered [ 1131.707681] Key type .llcrypt registered [ 1131.910615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.920678] alg: No test for adler32 (adler32-zlib) [ 1133.211624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.643240] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:18:00 (1788797880) [ 1139.730130] LNet: 55920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.739888] LNetError: 55920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1139.764933] LNet: Removed LNI 10.1.2.3@tcp [ 1140.379435] Key type .llcrypt unregistered [ 1140.384344] Key type ._llcrypt unregistered [ 1145.726158] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1145.990605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1145.994590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1145.999831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1148.703726] Key type ._llcrypt registered [ 1148.707432] Key type .llcrypt registered [ 1148.939467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.972399] alg: No test for adler32 (adler32-zlib) [ 1150.002068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1157.122885] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:18:18 (1788797898) [ 1158.737470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1160.663702] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:18:21 (1788797901) [ 1161.081395] LNet: 57109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.091522] LNetError: 57109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1161.110134] LNet: Removed LNI 10.1.2.3@tcp [ 1161.750531] Key type .llcrypt unregistered [ 1161.758477] Key type ._llcrypt unregistered [ 1168.270528] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1168.568575] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1168.579261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1168.586851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1171.399512] Key type ._llcrypt registered [ 1171.408189] Key type .llcrypt registered [ 1171.686210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.718827] alg: No test for adler32 (adler32-zlib) [ 1172.721828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.801097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1182.594822] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:18:43 (1788797923) [ 1183.328688] LNet: 58334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.343725] LNetError: 58334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1183.360044] LNet: Removed LNI 10.1.2.3@tcp [ 1183.958059] Key type .llcrypt unregistered [ 1183.959630] Key type ._llcrypt unregistered [ 1191.414570] Key type ._llcrypt registered [ 1191.416974] Key type .llcrypt registered [ 1191.659421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.677023] alg: No test for adler32 (adler32-zlib) [ 1192.674546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.746986] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1200.926801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1202.063209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_interrupt -r 1 [ 1202.126704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_interrupt -r 1 [ 1202.184152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1202.220732] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1202.663350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.704595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.777460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.854786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.928933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1203.016381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.039498] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1203.062993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.064356] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1203.112418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.165913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.203825] LNet: 58771:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.229713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.332785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.398699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.443473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.526670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.582697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.653859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1204.491729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_dropped -r 1 [ 1204.540659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_dropped -r 1 [ 1204.599146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1204.644409] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1204.934700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.970471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.998939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.032943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.064578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1205.111785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.157863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.210993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.240735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.258645] LNet: 58883:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.281933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.313519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.360709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.417874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.501542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.583941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.657734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1206.533269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_aborted -r 1 [ 1206.603983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_aborted -r 1 [ 1206.649775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1206.676871] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1207.015698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.087613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.128259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.162160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.195591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1207.227837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.261491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.298789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.335205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.354316] LNet: 58994:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.375166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.411408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.476766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.533522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.580699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.627555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.670490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1208.340632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_no_route -r 1 [ 1208.376568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_no_route -r 1 [ 1208.413671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1208.760230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.797274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.828655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.864445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.895673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1208.937582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1208.982270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.034617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.094620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.162043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.220554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.255902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.303737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.351486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.384517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.442551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.142964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_timeout -r 1 [ 1210.185085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_timeout -r 1 [ 1210.228943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1210.452994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.483148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.520141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.548446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.577749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1210.612872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.658555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.704177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.750487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.768155] LNet: 59216:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.780593] LNet: 59216:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1210.802603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.848024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.882502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.919746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.978262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.055546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.101073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1211.698548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_error -r 1 [ 1211.735801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_error -r 1 [ 1211.764202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1211.778213] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1211.781828] LNet: Skipped 2 previous similar messages [ 1211.991764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.022882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.053276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.086778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.117903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1212.162233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.205810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.238516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.264296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.292661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.327545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.365813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.397698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.428865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.461439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.520324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1212.914036] LNet: 59369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.936649] LNetError: 59369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.995418] LNet: Removed LNI 192.168.204.19@tcp [ 1214.450820] Key type .llcrypt unregistered [ 1214.456638] Key type ._llcrypt unregistered [ 1232.935876] Key type ._llcrypt registered [ 1232.938134] Key type .llcrypt registered [ 1234.123526] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:19:35 (1788797975) [ 1235.055402] Key type .llcrypt unregistered [ 1235.062666] Key type ._llcrypt unregistered [ 1242.293376] Key type ._llcrypt registered [ 1242.295879] Key type .llcrypt registered [ 1242.594288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.619582] alg: No test for adler32 (adler32-zlib) [ 1243.589793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1243.629937] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1250.994127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1261.968457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.014622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1262.044259] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1263.581332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_interrupt -r 1 [ 1263.627745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_interrupt -r 1 [ 1263.661234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_interrupt -r 1 [ 1263.695808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_interrupt -r 1 [ 1263.727540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_interrupt -r 1 [ 1263.760760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_interrupt -r 1 [ 1263.799630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_interrupt -r 1 [ 1263.839962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_interrupt -r 1 [ 1263.889178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1263.913405] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1263.922759] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1264.191317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.239871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.271974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.311574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.340821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1264.378331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.407937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.440304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.483606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.504747] LNet: 60832:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.523980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.556493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.594104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.643174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.709829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.773561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.816440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.411325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_dropped -r 1 [ 1265.475777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_dropped -r 1 [ 1265.537297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_dropped -r 1 [ 1265.618905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_dropped -r 1 [ 1265.669178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_dropped -r 1 [ 1265.724992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_dropped -r 1 [ 1265.775560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_dropped -r 1 [ 1265.805484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_dropped -r 1 [ 1265.852497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1265.872580] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1265.880898] LNet: Skipped 2 previous similar messages [ 1266.149970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.179250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1266.205152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.229479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.256640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1266.280834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.312737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.343191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.372397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.387730] LNet: 60955:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.402316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.430150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.449624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.478396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.499508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.539638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.566796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.096907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_aborted -r 1 [ 1267.130383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_aborted -r 1 [ 1267.166378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_aborted -r 1 [ 1267.219796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_aborted -r 1 [ 1267.285705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_aborted -r 1 [ 1267.341641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_aborted -r 1 [ 1267.391994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_aborted -r 1 [ 1267.453488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_aborted -r 1 [ 1267.506682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1267.539160] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1267.547749] LNet: Skipped 2 previous similar messages [ 1267.822728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.858573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.900961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.923920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.948726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1267.989028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.050877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.102676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.146850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.191643] LNet: 61078:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.243779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.296283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.332040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.367751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.434569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.501707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.558813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.227734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_no_route -r 1 [ 1269.270848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_no_route -r 1 [ 1269.326677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_no_route -r 1 [ 1269.358512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_no_route -r 1 [ 1269.397262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_no_route -r 1 [ 1269.440644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_no_route -r 1 [ 1269.490388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_no_route -r 1 [ 1269.540152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_no_route -r 1 [ 1269.575246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1269.598283] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1269.608253] LNet: Skipped 2 previous similar messages [ 1269.874865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.901104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.933228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.965713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.002292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1270.060197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.115923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.156170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.207640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.232487] LNet: 61202:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.252341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.280440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.337032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.382621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.411485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.451751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.494981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1271.002394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_timeout -r 1 [ 1271.041887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_timeout -r 1 [ 1271.075079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_timeout -r 1 [ 1271.122277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_timeout -r 1 [ 1271.155264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_timeout -r 1 [ 1271.184617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_timeout -r 1 [ 1271.233593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_timeout -r 1 [ 1271.286446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_timeout -r 1 [ 1271.327585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1271.657719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.707716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.752518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.779143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.814639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1271.841979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.870576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.899685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.927478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.953421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.981531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.017524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.052957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.081614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.107568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.134632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1272.689125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_error -r 1 [ 1272.717463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_error -r 1 [ 1272.754663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_error -r 1 [ 1272.792211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_error -r 1 [ 1272.831359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_error -r 1 [ 1272.866884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_error -r 1 [ 1272.923525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_error -r 1 [ 1272.973187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_error -r 1 [ 1273.028929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1273.302487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.330281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1273.381572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.411653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.436214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1273.463513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.515648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.558346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.623372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.685332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.712955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.744896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.791414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.826346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.860306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.892616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.937567] LNet: 61524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.944493] LNetError: 61524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.956230] LNet: Removed LNI 192.168.204.19@tcp [ 1279.966230] LNet: Removed LNI 192.168.204.19@tcp1 [ 1280.489620] Key type .llcrypt unregistered [ 1280.493568] Key type ._llcrypt unregistered [ 1296.751228] Key type ._llcrypt registered [ 1296.756338] Key type .llcrypt registered [ 1298.317345] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:20:39 (1788798039) [ 1299.360521] Key type .llcrypt unregistered [ 1299.363063] Key type ._llcrypt unregistered [ 1305.544683] Key type ._llcrypt registered [ 1305.550595] Key type .llcrypt registered [ 1305.772490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.799848] alg: No test for adler32 (adler32-zlib) [ 1306.779696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1306.818796] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1313.646935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1314.408457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_dropped -r 1 [ 1314.460451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_dropped -r 1 [ 1314.520245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1314.560030] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1314.794686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.825615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1314.856388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.880885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1314.902827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1314.931334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1314.953272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1314.979466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.003811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.018174] LNet: 62770:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.034606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.059395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.083927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.109389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.144796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.168560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.198417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.699558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_error -r 1 [ 1315.755314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_error -r 1 [ 1315.797653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1315.825647] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1316.102731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1316.136243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1316.164969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1316.206967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1316.237993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1316.275189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1316.318207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1316.360189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1316.397717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1316.422965] LNet: 62882:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.440578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1316.470815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1316.515583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.562233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1316.619779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.662847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1316.696419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.191451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_timeout -r 1 [ 1317.257676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_timeout -r 1 [ 1317.297225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1317.318981] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1317.523724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.548375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.577112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.607545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.635759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1317.660988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.693805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.725706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.760685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.782803] LNet: 62993:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.804433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1317.847594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1317.889855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.928524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.961457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.018087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.081934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1318.356514] LNet: 63034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.365656] LNetError: 63034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.381285] LNet: Removed LNI 192.168.204.19@tcp [ 1318.861457] Key type .llcrypt unregistered [ 1318.867431] Key type ._llcrypt unregistered [ 1335.613679] Key type ._llcrypt registered [ 1335.619811] Key type .llcrypt registered [ 1337.157679] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:21:17 (1788798077) [ 1337.998803] Key type .llcrypt unregistered [ 1338.003568] Key type ._llcrypt unregistered [ 1343.890270] Key type ._llcrypt registered [ 1343.892722] Key type .llcrypt registered [ 1344.264097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.305130] alg: No test for adler32 (adler32-zlib) [ 1345.295505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1345.331531] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1352.863762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1362.092570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.174640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1362.217560] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1363.821434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_dropped -r 1 [ 1363.895277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e remote_dropped -r 1 [ 1363.925217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_dropped -r 1 [ 1363.976315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e remote_dropped -r 1 [ 1364.014432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e remote_dropped -r 1 [ 1364.051388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e remote_dropped -r 1 [ 1364.103106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e remote_dropped -r 1 [ 1364.171518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e remote_dropped -r 1 [ 1364.243209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1364.280040] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1364.293516] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.119@tcp [ 1364.553784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1364.575783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1364.595611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.616307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.657798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1364.685780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.716832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.749030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.790355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.813708] LNet: 64499:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.829776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.866374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.894340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.923826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.956373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.986538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.018841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1365.049202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.579962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_error -r 1 [ 1365.615832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e remote_error -r 1 [ 1365.649283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_error -r 1 [ 1365.678640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e remote_error -r 1 [ 1365.713164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e remote_error -r 1 [ 1365.757791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e remote_error -r 1 [ 1365.787795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e remote_error -r 1 [ 1365.812892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e remote_error -r 1 [ 1365.843510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1365.863398] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1365.868490] LNet: Skipped 2 previous similar messages [ 1366.116457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1366.192857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.225606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.248797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.278278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1366.307467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.338564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.375774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.408962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.431117] LNet: 64624:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.448216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.477991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.505699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.531218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.563448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.592639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.627226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.646847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.083988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_timeout -r 1 [ 1367.118469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e remote_timeout -r 1 [ 1367.165399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_timeout -r 1 [ 1367.205361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e remote_timeout -r 1 [ 1367.263846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e remote_timeout -r 1 [ 1367.309484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e remote_timeout -r 1 [ 1367.351242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e remote_timeout -r 1 [ 1367.372347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e remote_timeout -r 1 [ 1367.399805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.119@tcp [ 1367.413926] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1367.419238] LNet: Skipped 1 previous similar message [ 1367.617517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.637373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.662761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.696486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.719850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1367.745223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.771282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.801737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.842789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.858046] LNet: 64749:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.879246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.900575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.918734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.960601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.991804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.030856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.083271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1368.114877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.291890] LNet: 64827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.301936] LNetError: 64827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.313731] LNet: Removed LNI 192.168.204.19@tcp [ 1373.330515] LNet: Removed LNI 192.168.204.19@tcp1 [ 1373.742808] Key type .llcrypt unregistered [ 1373.749685] Key type ._llcrypt unregistered [ 1393.507908] Key type ._llcrypt registered [ 1393.511593] Key type .llcrypt registered [ 1394.926349] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:22:15 (1788798135) [ 1396.663830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1398.383208] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:22:19 (1788798139) [ 1399.393305] Key type .llcrypt unregistered [ 1399.400123] Key type ._llcrypt unregistered [ 1406.783192] Key type ._llcrypt registered [ 1406.786621] Key type .llcrypt registered [ 1407.035356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1407.056504] alg: No test for adler32 (adler32-zlib) [ 1407.990991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1408.050748] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1416.207942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1416.885605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e network_timeout -r 1 [ 1416.960346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e network_timeout -r 1 [ 1417.008444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1417.037155] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1417.304619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1417.312191] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1417.345401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1417.375527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1417.415691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1417.451884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1417.486374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1417.511368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1417.551704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1417.595396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1417.610161] LNet: 66368:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1417.653553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1417.705735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1417.754342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1417.799399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1417.845035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1417.892655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1418.197775] LNet: 66407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.207786] LNetError: 66407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.230926] LNet: Removed LNI 192.168.204.19@tcp [ 1418.605661] Key type .llcrypt unregistered [ 1418.607390] Key type ._llcrypt unregistered [ 1438.765152] Key type ._llcrypt registered [ 1438.770156] Key type .llcrypt registered [ 1439.163673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1439.195981] alg: No test for adler32 (adler32-zlib) [ 1440.173348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1440.201078] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1449.307143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1459.334554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.413606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.467867] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1461.149611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e network_timeout -r 1 [ 1461.191863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e network_timeout -r 1 [ 1461.233870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e network_timeout -r 1 [ 1461.263274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e network_timeout -r 1 [ 1461.309907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e network_timeout -r 1 [ 1461.346484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e network_timeout -r 1 [ 1461.390404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e network_timeout -r 1 [ 1461.435649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e network_timeout -r 1 [ 1461.481159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1461.507472] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1461.729380] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1461.736694] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.119@tcp [ 1461.812860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1461.845836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1461.879547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1461.918950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1461.951508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1461.994740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1462.028143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1462.064682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1462.099823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1462.132770] LNet: 67285:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1462.168822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1462.214365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1462.249741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1462.274559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1462.301989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1462.326955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1467.189527] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1467.506130] LNet: 67359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.518148] LNetError: 67359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.523183] LNet: Removed LNI 192.168.204.19@tcp [ 1467.536281] LNet: Removed LNI 192.168.204.19@tcp1 [ 1467.919930] Key type .llcrypt unregistered [ 1467.926764] Key type ._llcrypt unregistered [ 1482.226095] Key type ._llcrypt registered [ 1482.231424] Key type .llcrypt registered [ 1483.529921] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:23:44 (1788798224) [ 1483.882343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1483.902641] alg: No test for adler32 (adler32-zlib) [ 1484.820428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.864812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.883829] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1485.504876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.527867] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1486.352743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 1486.415533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1486.450871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1486.475617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1486.502373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1486.553478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1486.586777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.19@tcp [ 1486.626964] LNetError: 68456:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.19@tcp to recovery [ 1486.880201] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1486.882741] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1488.932046] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1489.077643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.001180] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1492.612376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.645809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1492.711815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.727873] LNet: 68553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.733607] LNetError: 68553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.738628] LNet: Removed LNI 192.168.204.19@tcp [ 1492.754889] LNet: Removed LNI 192.168.204.19@tcp1 [ 1492.786061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.854885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.906874] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1493.711719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1493.745640] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1494.570947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 1494.622859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1494.689639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1494.742578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1494.775674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1494.804919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1494.835857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1494.869126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.19@tcp [ 1494.889566] LNetError: 68825:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.19@tcp to recovery [ 1495.903578] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1498.498811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1501.024863] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1501.031626] LNet: Skipped 1 previous similar message [ 1505.569310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1505.622369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1505.681704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.844959] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:24:14 (1788798254) [ 1514.026684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.089718] LNet: 69533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.097670] LNetError: 69533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.103361] LNetError: 69533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1514.109289] LNet: Removed LNI 192.168.204.19@tcp [ 1514.129401] LNet: Removed LNI 192.168.204.19@tcp1 [ 1514.174934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.276630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.303243] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1515.207274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1515.248295] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1516.148241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 1516.240043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1516.278277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1516.325540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1516.358283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1516.408494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1516.451571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.19@tcp --health 0 [ 1516.561948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.280323] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.19@tcp [ 1517.280419] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1517.293971] LNet: Skipped 1 previous similar message [ 1532.268394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1532.496542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1532.568855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1532.612552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.19@tcp --health 500 [ 1532.716995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.380684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.456769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.482452] LNet: 70025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.489155] LNetError: 70025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.491848] LNetError: 70025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1539.503543] LNet: Removed LNI 192.168.204.19@tcp [ 1539.516828] LNet: Removed LNI 192.168.204.19@tcp1 [ 1539.555442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.601807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.628363] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1540.493693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1540.514395] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1541.618451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 1541.689389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.782726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1541.861607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1541.929190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1541.988301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1542.031758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1542.062992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.19@tcp --health 0 [ 1543.713807] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1543.721897] LNet: Skipped 3 previous similar messages [ 1544.519678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1554.165553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1554.199063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1554.247546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1564.256910] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:25:04 (1788798304) [ 1564.861940] LNet: 71073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.870489] LNetError: 71073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.875684] LNetError: 71073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1564.880646] LNet: Removed LNI 192.168.204.19@tcp [ 1564.892156] LNet: Removed LNI 192.168.204.19@tcp1 [ 1565.734473] Key type .llcrypt unregistered [ 1565.743765] Key type ._llcrypt unregistered [ 1572.901137] Key type ._llcrypt registered [ 1572.913788] Key type .llcrypt registered [ 1573.171723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.209334] alg: No test for adler32 (adler32-zlib) [ 1574.084793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1574.119475] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1584.745813] Lustre: DEBUG MARKER: Initial discovery [ 1584.822136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.119@tcp [ 1587.751062] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1588.123878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1588.183763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.119@tcp [ 1588.203371] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1588.234131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1588.282132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.119@tcp -r 1 -m GET -l 3 [ 1588.345595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.119@tcp -r 1 -m GET -e local_error [ 1588.386480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.119@tcp -r 1 -m PUT -l 6 [ 1588.448118] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1590.012282] Lustre: DEBUG MARKER: Force oleg419-client.virtnet to discover 192.168.204.119@tcp (in background) [ 1590.316136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.119@tcp [ 1592.118616] Lustre: DEBUG MARKER: Force oleg419-server to discover 192.168.204.19@tcp [ 1593.439561] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1595.333563] Lustre: DEBUG MARKER: Wait for 71830 [ 1596.576337] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1597.495882] Lustre: DEBUG MARKER: Finished wait on 71830 [ 1597.546970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1597.590842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1597.935299] LNet: 72283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.943707] LNetError: 72283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.964552] LNet: Removed LNI 192.168.204.19@tcp [ 1598.269493] Key type .llcrypt unregistered [ 1598.273710] Key type ._llcrypt unregistered [ 1615.259320] Key type ._llcrypt registered [ 1615.262393] Key type .llcrypt registered [ 1616.841657] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:25:57 (1788798357) [ 1617.435368] Key type .llcrypt unregistered [ 1617.437725] Key type ._llcrypt unregistered [ 1624.219022] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1624.462518] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1624.470878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1624.478317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1626.957922] Key type ._llcrypt registered [ 1626.964360] Key type .llcrypt registered [ 1627.093914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1627.103629] alg: No test for adler32 (adler32-zlib) [ 1627.979950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.049651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.074330] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1628.645297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1628.662685] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1637.556441] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:26:18 (1788798378) [ 1637.661560] LNet: 74365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.666127] LNetError: 74365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1637.676084] LNet: Removed LNI 10.1.2.3@tcp [ 1637.708212] LNet: Removed LNI 192.168.204.19@tcp [ 1638.348726] Key type .llcrypt unregistered [ 1638.353211] Key type ._llcrypt unregistered [ 1645.747105] Key type ._llcrypt registered [ 1645.751407] Key type .llcrypt registered [ 1646.020376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1646.050830] alg: No test for adler32 (adler32-zlib) [ 1647.044908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.156956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.188654] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1647.944373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1647.977133] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1649.021606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.19@tcp [ 1650.145795] LNetError: Unexpected error -22 connecting to 192.168.204.19@tcp at host 192.168.204.19:988 [ 1655.548472] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:26:36 (1788798396) [ 1655.711963] LNet: 75497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.725988] LNetError: 75497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1655.740745] LNet: Removed LNI 10.1.2.3@tcp [ 1655.766645] LNet: Removed LNI 192.168.204.19@tcp [ 1656.456327] Key type .llcrypt unregistered [ 1656.458718] Key type ._llcrypt unregistered [ 1663.826849] Key type ._llcrypt registered [ 1663.834587] Key type .llcrypt registered [ 1664.113616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1664.140528] alg: No test for adler32 (adler32-zlib) [ 1665.179348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.259666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1665.304055] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1666.495198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1666.525919] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 1668.105882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.19@tcp1 --nid 192.168.204.19@tcp2 [ 1674.891851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp1 192.168.204.19@tcp2 [ 1674.917488] LNetError: 76333:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.19@tcp1 to 192.168.204.19@tcp2 - Route table may be misconfigured [ 1674.975519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp2 192.168.204.19@tcp1 [ 1683.331721] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:27:04 (1788798424) [ 1683.451988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.469695] LNet: 76900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.482456] LNetError: 76900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.490963] LNet: Removed LNI 192.168.204.19@tcp1 [ 1684.521271] LNet: Removed LNI 192.168.204.19@tcp2 [ 1684.565695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.641431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.700816] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1685.724651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.760229] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1686.991609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 1687.062934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.19@tcp [ 1687.124811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp1 192.168.204.19@tcp1 [ 1687.172330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.19@tcp -d 192.168.204.19@tcp -e network_timeout [ 1687.205215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -e network_timeout [ 1687.250465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -e network_timeout [ 1687.299786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -e network_timeout [ 1687.344858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1687.380849] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1687.405300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1687.558953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1687.647200] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1687.682217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1693.951683] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:27:15 (1788798435) [ 1694.359082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.410892] LNet: 77769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.418910] LNetError: 77769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.427714] LNetError: 77769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1694.448991] LNet: Removed LNI 192.168.204.19@tcp [ 1695.526217] LNet: Removed LNI 192.168.204.19@tcp1 [ 1695.566310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.666560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1695.998554] LNet: 77806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.436821] Key type .llcrypt unregistered [ 1696.438580] Key type ._llcrypt unregistered [ 1708.088823] Key type ._llcrypt registered [ 1708.092958] Key type .llcrypt registered [ 1709.268354] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:27:30 (1788798450) [ 1709.817496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.846673] alg: No test for adler32 (adler32-zlib) [ 1710.701227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.039710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1714.018792] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:27:35 (1788798455) [ 1714.246714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.266290] LNet: 78902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.282384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.353475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.379760] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1715.058285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1715.085986] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1716.032382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1716.060320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp1 [ 1716.097101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp1 [ 1721.751139] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:27:43 (1788798463) [ 1723.792619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg419-server [ 1725.501783] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:27:46 (1788798466) [ 1727.547532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg419-server [ 1729.228631] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:27:50 (1788798470) [ 1731.102389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg419-server [ 1732.786384] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:27:54 (1788798474) [ 1734.784819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg419-server [ 1736.200235] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:27:57 (1788798477) [ 1738.096720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg419-server [ 1740.095193] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:28:01 (1788798481) [ 1742.050512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg419-server [ 1743.950216] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:28:05 (1788798485) [ 1745.806231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg419-server [ 1747.351517] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:28:08 (1788798488) [ 1749.334692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg419-server [ 1750.961855] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:28:12 (1788798492) [ 1753.221524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg419-server [ 1754.778953] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:28:16 (1788798496) [ 1755.090079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.126601] LNet: 82780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.135038] LNetError: 82780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.153375] LNet: Removed LNI 192.168.204.19@tcp [ 1756.204507] LNet: Removed LNI 192.168.204.19@tcp1 [ 1756.248510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.314667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.351693] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1757.110097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1757.194162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1757.365503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.386287] LNet: 82928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.394865] LNetError: 82928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.400031] LNetError: 82928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1758.454886] LNet: Removed LNI 192.168.204.19@tcp [ 1758.517061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.606858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1758.634078] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1759.543252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1759.728269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.754912] LNet: 83073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.761091] LNetError: 83073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.776349] LNet: Removed LNI 192.168.204.19@tcp [ 1759.797472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.862228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1759.881050] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1760.507868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1760.608881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1760.818717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.895840] LNet: Removed LNI 192.168.204.19@tcp [ 1761.941211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.019142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1762.044414] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1762.780402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1762.976862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.000979] LNetError: 83366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.006397] LNetError: 83366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1764.071820] LNet: Removed LNI 192.168.204.19@tcp [ 1764.113673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.186273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.212395] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1764.905310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1764.982250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1765.183562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.232680] LNet: 83513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.245799] LNet: 83513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1766.310894] LNet: Removed LNI 192.168.204.19@tcp [ 1766.345548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.416174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1766.458245] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1767.170703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1767.350948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.364645] LNetError: 83658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.370043] LNetError: 83658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1767.386302] LNet: Removed LNI 192.168.204.19@tcp [ 1767.408825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.470512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.509961] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1768.328990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1768.458458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1768.698232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.768641] LNet: Removed LNI 192.168.204.19@tcp [ 1769.797707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.879889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1769.910073] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1770.752879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1771.007334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.070905] LNet: Removed LNI 192.168.204.19@tcp [ 1772.110658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.179925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.213676] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1773.068807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1773.158709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1774.613783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.643427] LNet: 84107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.654752] LNet: 84107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1775.717960] LNet: Removed LNI 192.168.204.19@tcp [ 1775.753847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.834798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1775.860732] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1776.650861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1776.940855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.962417] LNetError: 84253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.970607] LNetError: 84253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1776.989702] LNet: Removed LNI 192.168.204.19@tcp [ 1777.019782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.074370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.094214] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1777.726318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1777.801859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1778.057505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.111038] LNet: Removed LNI 192.168.204.19@tcp [ 1779.127762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.178828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1779.234668] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1780.015124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 1780.316954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.417455] LNet: Removed LNI 192.168.204.19@tcp [ 1781.440996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.474477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.490399] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1782.171708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1782.222742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.247728] LNet: Removed LNI 192.168.204.19@tcp [ 1782.274225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.358860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.393867] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1783.076577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1783.147664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.177916] LNet: Removed LNI 192.168.204.19@tcp [ 1783.215599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.268594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1783.343974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.374716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.417618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1783.503943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.545094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.604324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1783.733229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.773548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.834585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1783.857701] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1784.595882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.638539] LNet: Removed LNI 192.168.204.19@tcp [ 1784.678458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.743126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1791.605726] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:28:52 (1788798532) [ 1791.806769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.832455] LNet: 85566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.839490] LNet: 85566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1791.869724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.941549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1791.969342] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/180] [ 1793.240955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.298233] LNetError: 85704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.304644] LNetError: 85704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1793.318716] LNet: Removed LNI 192.168.204.19@tcp231 [ 1793.342943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.403990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1793.427232] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/180] [ 1793.432586] LNet: 85712:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1793.916855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1793.978737] LNet: Removed LNI 192.168.204.19@tcp231 [ 1794.006996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1794.054552] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/180] [ 1795.163050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.203921] LNet: Removed LNI 192.168.204.19@tcp231 [ 1795.233181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.299845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1795.323261] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/0] [ 1795.328830] LNet: 85864:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1795.653630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1795.687348] LNet: Removed LNI 192.168.204.19@tcp231 [ 1795.706145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1795.730867] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/0] [ 1797.067721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.108139] LNet: Removed LNI 192.168.204.19@tcp231 [ 1797.139844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.209395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1797.242788] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/60] [ 1797.250261] LNet: 86016:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1797.825305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1797.862123] LNet: Removed LNI 192.168.204.19@tcp231 [ 1797.910631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1797.956287] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/60] [ 1799.282809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.324179] LNet: Removed LNI 192.168.204.19@tcp231 [ 1799.353533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.454712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1799.483920] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/180] [ 1799.490406] LNet: 86167:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1799.870725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1799.919945] LNet: Removed LNI 192.168.204.19@tcp231 [ 1799.961771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1800.012210] LNet: Added LNI 192.168.204.19@tcp231 [8/256/0/180] [ 1808.470848] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:29:09 (1788798549) [ 1808.700579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.725528] LNet: Removed LNI 192.168.204.19@tcp231 [ 1808.738376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.789473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1808.806570] LNet: Added LNI 192.168.204.19@tcp232 [8/256/0/180] [ 1809.544035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1809.595383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1809.640746] LNet: Removed LNI 192.168.204.19@tcp232 [ 1809.672804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1809.721502] LNet: Added LNI 192.168.204.19@tcp232 [8/256/0/180] [ 1810.478565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1810.513528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1810.555978] LNet: Removed LNI 192.168.204.19@tcp232 [ 1810.569892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1810.595887] LNet: Added LNI 192.168.204.19@tcp232 [8/256/0/180] [ 1811.208730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1811.246753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1811.283711] LNet: Removed LNI 192.168.204.19@tcp232 [ 1811.314456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1811.349945] LNet: Added LNI 192.168.204.19@tcp232 [8/256/0/180] [ 1812.075234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1812.121893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1812.155915] LNet: Removed LNI 192.168.204.19@tcp232 [ 1812.180773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1818.425380] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:29:19 (1788798559) [ 1819.379635] Key type .llcrypt unregistered [ 1819.381845] Key type ._llcrypt unregistered [ 1825.020451] Key type ._llcrypt registered [ 1825.025693] Key type .llcrypt registered [ 1825.188211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.214564] alg: No test for adler32 (adler32-zlib) [ 1826.069953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1826.097164] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1832.254597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1840.006679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.043630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1840.069159] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1841.220446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_interrupt -r 2 [ 1841.265865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_interrupt -r 2 [ 1841.319955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_interrupt -r 2 [ 1841.356034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_interrupt -r 2 [ 1841.392368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_interrupt -r 2 [ 1841.427887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_interrupt -r 2 [ 1841.453118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_interrupt -r 2 [ 1841.492698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_interrupt -r 2 [ 1841.522360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1841.543671] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1841.549986] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1841.786570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1841.842303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1841.897365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.921371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1841.955363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1842.001762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1842.036073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1842.063318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1842.126257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1842.151073] LNet: 88665:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1842.171693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1842.202484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1842.233435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1842.259346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1842.296865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1842.334544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1842.850975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_dropped -r 2 [ 1842.877515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_dropped -r 2 [ 1842.908464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_dropped -r 2 [ 1842.940699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_dropped -r 2 [ 1842.981984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_dropped -r 2 [ 1843.012243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_dropped -r 2 [ 1843.043304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_dropped -r 2 [ 1843.076488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_dropped -r 2 [ 1843.111334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1843.129225] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1843.385284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1843.419817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1843.453116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1843.483891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1843.518825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1843.549944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1843.580197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1843.604163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1843.623309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1843.634385] LNet: 88787:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1843.645583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1843.672529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1843.691715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1843.712758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1843.735541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1843.769263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1844.271606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_aborted -r 2 [ 1844.357229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_aborted -r 2 [ 1844.394518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_aborted -r 2 [ 1844.445129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_aborted -r 2 [ 1844.473311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_aborted -r 2 [ 1844.526109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_aborted -r 2 [ 1844.588151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_aborted -r 2 [ 1844.625237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_aborted -r 2 [ 1844.650244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1844.663836] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1844.993601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1845.043692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1845.071388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1845.088687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1845.127308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1845.160930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1845.188842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1845.234280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1845.287719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1845.307267] LNet: 88909:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1845.336677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1845.360939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1845.383103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1845.409442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1845.441188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1845.470742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1845.986155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_no_route -r 2 [ 1846.019895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_no_route -r 2 [ 1846.061188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_no_route -r 2 [ 1846.101573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_no_route -r 2 [ 1846.159977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_no_route -r 2 [ 1846.226450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_no_route -r 2 [ 1846.263976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_no_route -r 2 [ 1846.299220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_no_route -r 2 [ 1846.336516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1846.632752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1846.671280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1846.704389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1846.744919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1846.783300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1846.834685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1846.876191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1846.915756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.967130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1847.006094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1847.051334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1847.093502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1847.113791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1847.147556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1847.195954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1847.787641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e local_timeout -r 2 [ 1847.852468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e local_timeout -r 2 [ 1847.898330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e local_timeout -r 2 [ 1847.931480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e local_timeout -r 2 [ 1847.953559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e local_timeout -r 2 [ 1847.990880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e local_timeout -r 2 [ 1848.043259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e local_timeout -r 2 [ 1848.083682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e local_timeout -r 2 [ 1848.118415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1848.138903] LNet: There was an unexpected network error while writing to 192.168.204.119: rc = -22 [ 1848.143701] LNet: Skipped 1 previous similar message [ 1848.375332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1848.407506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1848.439834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1848.466487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1848.489719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1848.513337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1848.535885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1848.561089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1848.605799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1848.635856] LNet: 89154:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1848.645067] LNet: 89154:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1848.662667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1848.763061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1848.831799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1848.903140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1848.939332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1848.993196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1849.360275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp -m GET -e remote_dropped -r 2 [ 1849.405678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.119@tcp1 -m GET -e remote_dropped -r 2 [ 1849.462699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp -m GET -e remote_dropped -r 2 [ 1849.500782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp -d 192.168.204.19@tcp1 -m GET -e remote_dropped -r 2 [ 1849.529322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp -m GET -e remote_dropped -r 2 [ 1849.567673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.119@tcp1 -m GET -e remote_dropped -r 2 [ 1849.608668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp -m GET -e remote_dropped -r 2 [ 1849.648441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.19@tcp1 -d 192.168.204.19@tcp1 -m GET -e remote_dropped -r 2 [ 1849.700301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.19@tcp 192.168.204.119@tcp [ 1849.719302] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.119@tcp [ 1849.972663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1850.013503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1850.042976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1850.072817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1850.104488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1850.127733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1850.154146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1850.179823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1850.223906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1850.270260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1850.346715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1850.405950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1850.440660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1850.493895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1850.531903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1857.204341] LNet: 89350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.219930] LNetError: 89350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.230409] LNet: Removed LNI 192.168.204.19@tcp [ 1857.245455] LNet: Removed LNI 192.168.204.19@tcp1 [ 1857.698160] Key type .llcrypt unregistered [ 1857.700153] Key type ._llcrypt unregistered [ 1878.447476] Key type ._llcrypt registered [ 1878.449604] Key type .llcrypt registered [ 1879.754850] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:30:20 (1788798620) [ 1880.247314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1880.272380] alg: No test for adler32 (adler32-zlib) [ 1881.273814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.321558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.370542] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1882.290379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1882.341402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.19@tcp [ 1882.394957] LNetError: 90328:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.19@tcp to recovery [ 1882.497561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.19@tcp [ 1882.618114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1888.842964] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:30:30 (1788798630) [ 1889.022838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.064288] LNet: 90914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.067889] LNetError: 90914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.082600] LNet: Removed LNI 192.168.204.19@tcp [ 1889.102844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.135687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.160854] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1889.997835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1890.022833] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 1891.057716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.19@tcp [ 1891.084947] LNetError: 91184:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.19@tcp to recovery [ 1891.171624] LNet: 1 local NIs in recovery (showing 1): 192.168.204.19@tcp [ 1891.178992] LNet: There was an unexpected network error while writing to 192.168.204.19: rc = -22 [ 1892.348954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.19@tcp [ 1899.682339] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:30:40 (1788798640) [ 1901.515651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg419-server [ 1903.081750] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:30:44 (1788798644) [ 1904.502811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg419-server [ 1906.789275] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:30:47 (1788798647) [ 1909.881513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg419-server [ 1912.215304] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:30:53 (1788798653) [ 1912.667463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.741414] LNet: 92806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.751981] LNetError: 92806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.767460] LNet: Removed LNI 192.168.204.19@tcp [ 1912.786107] LNet: Removed LNI 192.168.204.19@tcp1 [ 1912.820463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.872691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.903448] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1913.941782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1914.116962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1914.208826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1914.386892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1914.419798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1914.491073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1922.529622] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:31:03 (1788798663) [ 1922.760257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.778291] LNet: 93537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.786593] LNetError: 93537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.793248] LNetError: 93537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1922.806591] LNet: Removed LNI 192.168.204.19@tcp [ 1922.827524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.889546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.917793] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1930.310544] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:31:11 (1788798671) [ 1930.459741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.512955] LNet: 94235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.529216] LNetError: 94235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.579400] LNet: Removed LNI 192.168.204.19@tcp [ 1930.622539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.730264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.749154] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1937.842800] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:31:18 (1788798678) [ 1939.286556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1941.326983] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:31:22 (1788798682) [ 1941.997197] LNet: 95276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.006362] LNetError: 95276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.022881] LNet: Removed LNI 192.168.204.19@tcp [ 1942.758814] Key type .llcrypt unregistered [ 1942.763835] Key type ._llcrypt unregistered [ 1949.028478] Key type ._llcrypt registered [ 1949.031170] Key type .llcrypt registered [ 1949.177937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1949.191459] alg: No test for adler32 (adler32-zlib) [ 1950.100051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1950.125561] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1956.961173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 1961.603395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.119@tcp [ 1961.888317] LNet: 95690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.900300] LNetError: 95690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.936226] LNet: Removed LNI 192.168.204.19@tcp [ 1962.378654] Key type .llcrypt unregistered [ 1962.382391] Key type ._llcrypt unregistered [ 1973.054629] Key type ._llcrypt registered [ 1973.056718] Key type .llcrypt registered [ 1974.521407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1976.021923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1977.688219] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:31:58 (1788798718) [ 1979.065315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1980.578433] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:32:01 (1788798721) [ 1982.138407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg419-server [ 1983.710367] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:32:05 (1788798725) [ 1985.307345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg419-server [ 1986.880792] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:32:08 (1788798728) [ 1988.255481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.312500] alg: No test for adler32 (adler32-zlib) [ 1989.538041] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1989.854619] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 1991.559343] Key type lgssc registered [ 1992.835362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.002874] Key type lgssc unregistered [ 2016.311295] LNet: 98861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.320581] LNetError: 98861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.346916] LNet: Removed LNI 192.168.204.19@tcp [ 2017.128570] Key type .llcrypt unregistered [ 2017.130926] Key type ._llcrypt unregistered [ 2031.863887] Key type ._llcrypt registered [ 2031.867616] Key type .llcrypt registered [ 2033.513782] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:32:54 (1788798774) [ 2033.940771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.963344] alg: No test for adler32 (adler32-zlib) [ 2034.921644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.996941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.074517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.134364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.192140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2041.499662] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:33:02 (1788798782) [ 2041.681873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.702669] LNet: 100277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.728701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.775434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.809572] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2042.631170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2042.656898] LNet: 100423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.674476] LNetError: 100423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.688885] LNet: Removed LNI 192.168.204.19@tcp [ 2042.712537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.781538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2042.812452] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2043.345829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.380878] LNetError: 100451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.405245] LNet: Removed LNI 192.168.204.19@tcp [ 2043.421519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.482935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2043.508367] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2050.634924] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:33:11 (1788798791) [ 2050.752615] LNet: 101033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.762721] LNet: 101033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2050.772465] LNetError: 101033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.794303] LNet: Removed LNI 192.168.204.19@tcp [ 2051.425729] Key type .llcrypt unregistered [ 2051.427390] Key type ._llcrypt unregistered [ 2057.381271] Key type ._llcrypt registered [ 2057.383766] Key type .llcrypt registered [ 2057.581823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2057.611318] alg: No test for adler32 (adler32-zlib) [ 2058.544239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.606620] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2058.608675] LNet: *** cfs_fail_loc=e003, val=100*** [ 2059.142267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2059.158565] LNet: *** cfs_fail_loc=e003, val=100*** [ 2059.161500] LNet: Skipped 2 previous similar messages [ 2059.166382] Ping reply received with mapping: [ 2059.169587] found metadata reply NI[192.168.204.19@tcp] [ 2059.172179] 13 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2059.175822] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2059.177937] 5a 5a 5a 5a 5a 5a 5a 5a [ 2059.279564] LNet: 101449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.285544] LNetError: 101449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.301812] LNet: Removed LNI 192.168.204.19@tcp [ 2059.839710] Key type .llcrypt unregistered [ 2059.842839] Key type ._llcrypt unregistered [ 2070.275260] Key type ._llcrypt registered [ 2070.277259] Key type .llcrypt registered [ 2071.210324] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:33:32 (1788798812) [ 2071.687336] Key type .llcrypt unregistered [ 2071.688915] Key type ._llcrypt unregistered [ 2077.104451] Key type ._llcrypt registered [ 2077.107755] Key type .llcrypt registered [ 2077.380038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.427212] alg: No test for adler32 (adler32-zlib) [ 2078.501339] LNetError: 102485:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2078.910162] Key type .llcrypt unregistered [ 2078.914312] Key type ._llcrypt unregistered [ 2083.327141] Key type ._llcrypt registered [ 2083.336569] Key type .llcrypt registered [ 2084.484182] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:33:45 (1788798825) [ 2084.969431] Key type .llcrypt unregistered [ 2084.974182] Key type ._llcrypt unregistered [ 2090.305063] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2090.307654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2092.780844] Key type ._llcrypt registered [ 2092.785290] Key type .llcrypt registered [ 2092.960079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.978675] alg: No test for adler32 (adler32-zlib) [ 2093.967088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.026546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2102.608248] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:34:04 (1788798844) [ 2102.837904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.873445] LNet: 104259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.896337] LNetError: 104259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2102.908241] LNet: Removed LNI 10.1.2.3@tcp [ 2102.953554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.431910] Key type .llcrypt unregistered [ 2103.434501] Key type ._llcrypt unregistered [ 2108.165602] Key type ._llcrypt registered [ 2108.167864] Key type .llcrypt registered [ 2108.279282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2108.294728] alg: No test for adler32 (adler32-zlib) [ 2109.239394] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2109.325916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2109.554365] LNet: 104544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.566872] LNetError: 104544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.596570] LNet: Removed LNI 192.168.204.19@tcp [ 2110.252472] Key type .llcrypt unregistered [ 2110.255546] Key type ._llcrypt unregistered [ 2123.833191] Key type ._llcrypt registered [ 2123.839109] Key type .llcrypt registered [ 2126.015598] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:34:26 (1788798866) [ 2127.454339] Key type .llcrypt unregistered [ 2127.455882] Key type ._llcrypt unregistered [ 2135.098966] Key type ._llcrypt registered [ 2135.101361] Key type .llcrypt registered [ 2135.244270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2135.255496] alg: No test for adler32 (adler32-zlib) [ 2142.301992] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:34:43 (1788798883) [ 2142.584576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.623269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.680504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.719198] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2143.501655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.542819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2143.591390] LNet: Removed LNI 192.168.204.19@tcp [ 2143.939887] LNet: 106283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.397639] Key type .llcrypt unregistered [ 2144.400182] Key type ._llcrypt unregistered [ 2155.122896] Key type ._llcrypt registered [ 2155.128270] Key type .llcrypt registered [ 2156.447037] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:34:57 (1788798897) [ 2156.748172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.786742] alg: No test for adler32 (adler32-zlib) [ 2157.768477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.821359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.845566] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2158.611718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2158.657332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.19@tcp [ 2158.681728] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2158.685691] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2158.688850] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2158, 0, 0 [ 2158.693654] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2158.696881] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2158.700616] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2158.703064] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2158, 0, 0 [ 2158.707808] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2158.710203] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2158.713663] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2158.716509] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2158, 0, 2208 [ 2158.725471] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2158.731371] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2158.734819] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2158.737438] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2158, 0, 2208 [ 2158.741189] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2165.755807] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:35:06 (1788798906) [ 2166.265466] LNet: 107820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.281065] LNetError: 107820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.298395] LNet: Removed LNI 192.168.204.19@tcp [ 2167.073743] Key type .llcrypt unregistered [ 2167.076539] Key type ._llcrypt unregistered [ 2173.852409] Key type ._llcrypt registered [ 2173.853950] Key type .llcrypt registered [ 2174.132849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2174.163603] alg: No test for adler32 (adler32-zlib) [ 2175.077835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2175.101389] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2182.617852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 2190.612536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.670545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2190.692685] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 2191.906985] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2192.083802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2192.086779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2192.092247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2194.440165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2194.475251] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2203.425449] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:35:44 (1788798944) [ 2203.553581] LNet: 109319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.564797] LNetError: 109319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.571410] LNet: Removed LNI 192.168.204.19@tcp [ 2204.575551] LNet: Removed LNI 192.168.204.19@tcp1 [ 2204.577455] LNetError: 109319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2204.589052] LNetError: 109319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2204.612024] LNet: Removed LNI 10.1.2.3@tcp99 [ 2205.186901] Key type .llcrypt unregistered [ 2205.192477] Key type ._llcrypt unregistered [ 2210.280269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.286115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2212.792424] Key type ._llcrypt registered [ 2212.794218] Key type .llcrypt registered [ 2212.976971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2212.999743] alg: No test for adler32 (adler32-zlib) [ 2213.953673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.020750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.054735] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2214.618983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2214.638903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2216.851228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2216.907960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2216.938806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2216.958527] LNetError: 110064:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2216.991199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2217.043436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2217.543258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2217.685483] LNet: 110084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.704156] LNetError: 110084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2217.713456] LNet: Removed LNI 10.1.2.3@tcp [ 2218.729492] LNet: Removed LNI 192.168.204.19@tcp [ 2219.450717] Key type .llcrypt unregistered [ 2219.455480] Key type ._llcrypt unregistered [ 2229.762713] Key type ._llcrypt registered [ 2229.764993] Key type .llcrypt registered [ 2231.036556] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:36:12 (1788798972) [ 2231.430682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.453261] alg: No test for adler32 (adler32-zlib) [ 2232.404545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.496973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.525631] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2233.321433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2233.398974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg419-client@tcp [ 2239.150063] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:36:20 (1788798980) [ 2239.293437] LNet: 111605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.301659] LNetError: 111605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.315142] LNet: Removed LNI 192.168.204.19@tcp [ 2240.103672] Key type .llcrypt unregistered [ 2240.105705] Key type ._llcrypt unregistered [ 2245.691420] Key type ._llcrypt registered [ 2245.693677] Key type .llcrypt registered [ 2245.870219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.893636] alg: No test for adler32 (adler32-zlib) [ 2246.814622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2246.839692] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2246.873576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2251.887874] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:36:33 (1788798993) [ 2253.374824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2254.910461] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:36:36 (1788798996) [ 2256.317535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2258.034593] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:36:39 (1788798999) [ 2258.102996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.147077] LNet: 113025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.165474] LNetError: 113025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.186879] LNet: Removed LNI 192.168.204.19@tcp [ 2258.219263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.269683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.316050] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2259.039652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2259.080073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2265.538659] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:36:46 (1788799006) [ 2265.761459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.792563] LNet: 113741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.798491] LNetError: 113741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.830295] LNet: Removed LNI 192.168.204.19@tcp [ 2265.865066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.933586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2265.976855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2266.526556] Key type .llcrypt unregistered [ 2266.532537] Key type ._llcrypt unregistered [ 2276.994653] Key type ._llcrypt registered [ 2276.997211] Key type .llcrypt registered [ 2278.490747] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:36:59 (1788799019) [ 2278.874941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.887959] alg: No test for adler32 (adler32-zlib) [ 2279.843494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.888611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.906799] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2280.578986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2280.644887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2287.110361] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:37:08 (1788799028) [ 2287.223919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.253870] LNet: 115287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.267290] LNetError: 115287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.288484] LNet: Removed LNI 192.168.204.19@tcp [ 2287.341439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.407323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2287.453857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2293.238552] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:37:14 (1788799034) [ 2293.541777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.572680] LNet: 115873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.588538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.661540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2294.132834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2294.171332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2294.233088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2294.667742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2294.720713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2294.814648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.851097] LNet: 115906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.878409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.942861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2295.478954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2295.513412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2295.582723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2296.016851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2296.068626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2302.591509] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:37:23 (1788799043) [ 2302.779259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.802900] LNet: 116506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.831342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.907109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2302.930882] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2303.595198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2303.625276] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2304.492834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2304.562554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2304.853674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.882708] LNetError: 116786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.951181] LNet: Removed LNI 192.168.204.19@tcp [ 2305.953718] LNetError: 116786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.975514] LNet: Removed LNI 192.168.204.19@tcp2 [ 2306.017140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.107974] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2306.113486] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2306.120982] LNet: 116792:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2306.520409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2306.710340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.743182] LNet: Removed LNI 192.168.204.19@tcp [ 2307.762212] LNet: Removed LNI 192.168.204.19@tcp2 [ 2307.794261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.860558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.886656] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2308.548834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2308.564831] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2309.417989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2309.477706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2309.531264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2309.726811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.743956] LNet: 117114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.747861] LNet: 117114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2309.753832] LNetError: 117114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.761523] LNetError: 117114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2309.767602] LNet: Removed LNI 192.168.204.19@tcp [ 2310.822242] LNet: Removed LNI 192.168.204.19@tcp2 [ 2310.844198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.899901] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2310.908447] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2310.914460] LNet: 117120:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2311.256531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2311.444637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.466455] LNet: Removed LNI 192.168.204.19@tcp [ 2312.495342] LNet: Removed LNI 192.168.204.19@tcp2 [ 2312.536212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.604864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.641491] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2313.316333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2313.333639] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2314.074865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2314.123701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2314.327545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.360096] LNetError: 117436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.366871] LNetError: 117436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2314.373609] LNet: Removed LNI 192.168.204.19@tcp [ 2314.383824] LNet: Removed LNI 192.168.204.19@tcp2 [ 2314.413550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.495119] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2314.507035] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2314.514114] LNet: 117443:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2314.959373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2315.165074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.192305] LNet: Removed LNI 192.168.204.19@tcp [ 2316.207973] LNet: Removed LNI 192.168.204.19@tcp2 [ 2316.232106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.298898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.338738] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2316.941048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2316.965056] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2317.832576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2317.868562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2317.923620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2318.171319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.196318] LNet: 117765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.202571] LNet: 117765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2319.264909] LNet: Removed LNI 192.168.204.19@tcp [ 2320.296369] LNet: Removed LNI 192.168.204.19@tcp2 [ 2320.331613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.413634] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2320.420790] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2320.425768] LNet: 117771:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2320.892572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2321.089769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.113248] LNet: Removed LNI 192.168.204.19@tcp [ 2322.159599] LNet: Removed LNI 192.168.204.19@tcp2 [ 2322.209406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.331794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.366037] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2323.140333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2323.170335] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2323.965964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2324.003351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2324.069882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2324.337030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.359902] LNetError: 118105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.366507] LNetError: 118105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2324.383231] LNet: Removed LNI 192.168.204.19@tcp [ 2324.400123] LNet: Removed LNI 192.168.204.19@tcp2 [ 2324.443439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.495356] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2324.499900] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2324.504712] LNet: 118111:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2325.007763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2325.403181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.431155] LNet: Removed LNI 192.168.204.19@tcp [ 2326.441875] LNet: Removed LNI 192.168.204.19@tcp2 [ 2326.480797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.580644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.608897] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2327.238741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2327.268432] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2328.048920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2328.095995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2328.124972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2328.184220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2328.569377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.631143] LNet: Removed LNI 192.168.204.19@tcp [ 2330.663556] LNet: Removed LNI 192.168.204.19@tcp2 [ 2330.706370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.794264] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2330.803544] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2331.349234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2337.655974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2339.005599] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:38:00 (1788799080) [ 2339.295562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.341297] LNet: 119230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.349736] LNet: 119230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2339.356230] LNet: Removed LNI 192.168.204.19@tcp [ 2340.390391] LNet: Removed LNI 192.168.204.19@tcp2 [ 2340.421526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.699621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2340.721725] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2341.478877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2341.497919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2342.298939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.322684] LNetError: 119503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2342.327962] LNetError: 119503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2342.333748] LNet: Removed LNI 10.1.2.3@tcp [ 2342.342754] LNet: Removed LNI 192.168.204.19@tcp [ 2342.362563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.430211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2342.451405] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2342.454556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2342.816336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2342.843968] LNet: Removed LNI 192.168.204.19@tcp [ 2342.863682] LNet: Removed LNI 10.1.2.3@tcp [ 2349.879170] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:38:10 (1788799090) [ 2350.299512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.354884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.434255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.466718] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2351.359973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2351.421688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2351.514432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.575900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.626777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.691159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.766660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.810185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.866738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.906255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.942597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2351.993125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2352.247889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2358.148561] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:38:19 (1788799099) [ 2358.258644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.321897] LNet: Removed LNI 192.168.204.19@tcp [ 2358.383812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.462540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.495103] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2359.367124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2359.398843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2359.463474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.528970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.589504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.630970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.663715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.705866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.771132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.823862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.881731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2359.931908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.042581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2360.079883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.116163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.170318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.225552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.282041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.341620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.402431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.467690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.539417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2360.602134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.19@tcp [ 2366.791984] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:38:27 (1788799107) [ 2366.877964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.978326] LNet: Removed LNI 192.168.204.19@tcp [ 2368.031141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.270631] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2368.530779] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2368.536298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2368.553762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2370.888800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2370.924588] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2371.564640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2371.585138] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2374.839856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2374.937946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2375.077263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.19@tcp -m PUT -e local_timeout -r 1 [ 2375.191026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2375.282246] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2375.291467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2375.581000] LNetError: 121670:0:(lib-move.c:4794:lnet_parse()) 192.168.204.19@tcp, src 192.168.204.19@tcp: Dropping PUT (error -108 looking up sender) [ 2376.607887] LNet: 122128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.612891] LNet: 122128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2376.638854] LNetError: 122128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2376.655092] LNetError: 122128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2376.671208] LNet: Removed LNI 10.1.2.3@tcp [ 2377.704395] LNet: Removed LNI 192.168.204.19@tcp [ 2378.343209] Key type .llcrypt unregistered [ 2378.345357] Key type ._llcrypt unregistered [ 2391.271975] Key type ._llcrypt registered [ 2391.274879] Key type .llcrypt registered [ 2392.731727] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:38:53 (1788799133) [ 2393.269645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2393.286685] alg: No test for adler32 (adler32-zlib) [ 2394.216160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.656648] LNet: 122978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.295337] Key type .llcrypt unregistered [ 2395.300515] Key type ._llcrypt unregistered [ 2402.080076] Key type ._llcrypt registered [ 2402.082167] Key type .llcrypt registered [ 2402.354447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2402.375264] alg: No test for adler32 (adler32-zlib) [ 2403.437426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2403.478458] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2411.482745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 2411.699813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2412.358531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.119@tcp [ 2412.652084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2412.696423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2412.703128] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.119@tcp [ 2412.748272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2412.826809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2412.890572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2412.968786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2413.029219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2413.078790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2413.122493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2413.140882] LNet: 123428:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2413.169421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2413.261467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2413.310038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2413.361467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2413.418958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2413.463558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2414.215572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2414.518440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2414.543927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2414.568146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2414.611249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2414.648356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.671750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.701859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.735803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.770456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.790268] LNet: 123547:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.809811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.845299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.898291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.936079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.978120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2415.038859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.773466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.119@tcp [ 2416.111082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.159764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.199772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.249906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.298681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2416.347321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.394930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.441920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.490496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.513960] LNet: 123666:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.537393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.588307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.622910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.651734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.706511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.749874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.361580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2417.568531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2417.597945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2417.622910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2417.645710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2417.675523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2417.708243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2417.733453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2417.761555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2417.788506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2417.815542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2417.863423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2417.905354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2417.937665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2417.994982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.042415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2418.394519] LNet: 123832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.406547] LNetError: 123832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2418.429663] LNet: Removed LNI 192.168.204.19@tcp [ 2418.863919] Key type .llcrypt unregistered [ 2418.869031] Key type ._llcrypt unregistered [ 2435.598092] Key type ._llcrypt registered [ 2435.599522] Key type .llcrypt registered [ 2437.068937] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:39:37 (1788799177) [ 2437.658954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.680258] alg: No test for adler32 (adler32-zlib) [ 2438.639665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2439.079551] LNet: 124717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.818439] Key type .llcrypt unregistered [ 2439.829560] Key type ._llcrypt unregistered [ 2445.869671] Key type ._llcrypt registered [ 2445.873031] Key type .llcrypt registered [ 2446.173980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2446.186704] alg: No test for adler32 (adler32-zlib) [ 2447.085152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2447.119696] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2455.015057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.119@tcp [ 2465.093254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.158463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2465.193505] LNet: Added LNI 192.168.204.19@tcp1 [8/256/0/180] [ 2466.267237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2467.019510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.119@tcp [ 2467.302714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2467.339457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2467.387956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2467.436687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2467.501724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2467.578942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2467.616297] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.119@tcp [ 2467.647642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2467.693869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2467.768586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2467.823806] LNet: 125369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2467.889647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2467.966192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2468.014676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2468.072288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2468.109938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2468.150439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2469.193657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2469.569564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2469.616973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2469.662727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2469.712771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2469.785968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2469.853986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2469.921703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2470.013597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2470.063915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2470.104762] LNet: 125489:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2470.130649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2470.157853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2470.229114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2470.281923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2470.352782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2470.395491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2471.261328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.119@tcp [ 2471.554377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2471.590495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2471.626531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2471.664640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2471.707343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2471.752723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2471.789541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2471.819238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2471.860861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2471.882378] LNet: 125608:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2471.901803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2471.927328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2471.954807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2471.983727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2472.009724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2472.038228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2472.674832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2472.941808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2472.975952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2473.011618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2473.076485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2473.131816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2473.181820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2473.242050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2473.290441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2473.323536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2473.355491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2473.399911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2473.439368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2473.478888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2473.523177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2473.561621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.954938] LNet: 125809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.984518] LNetError: 125809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.992830] LNet: Removed LNI 192.168.204.19@tcp [ 2479.003698] LNet: Removed LNI 192.168.204.19@tcp1 [ 2479.561901] Key type .llcrypt unregistered [ 2479.572742] Key type ._llcrypt unregistered [ 2496.357258] Key type ._llcrypt registered [ 2496.360454] Key type .llcrypt registered [ 2497.786605] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:40:38 (1788799238) [ 2498.230954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.255122] alg: No test for adler32 (adler32-zlib) [ 2499.210974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2499.275874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2499.311808] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2500.091697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2500.116209] LNet: Added LNI 192.168.204.19@tcp2 [8/256/0/180] [ 2501.275462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.19@tcp [ 2501.401891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2502.431159] LNet: Removed LNI 192.168.204.19@tcp [ 2509.215378] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:40:50 (1788799250) [ 2511.002991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2513.141033] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:40:53 (1788799253) [ 2521.507348] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:41:02 (1788799262) [ 2529.366037] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:41:10 (1788799270) [ 2537.729229] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:41:18 (1788799278) [ 2547.091299] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:41:27 (1788799287) [ 2554.480393] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:41:35 (1788799295) [ 2562.224137] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:41:43 (1788799303) [ 2570.837038] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:41:51 (1788799311) [ 2579.005344] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:41:59 (1788799319) [ 2586.947159] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:42:08 (1788799328) [ 2594.401249] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:42:15 (1788799335) [ 2601.691436] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:42:23 (1788799343) [ 2608.369304] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:42:29 (1788799349) [ 2615.103343] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:42:36 (1788799356) [ 2620.609313] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:42:42 (1788799362) [ 2626.608112] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:42:48 (1788799368) [ 2633.080381] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:42:54 (1788799374) [ 2640.000969] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:43:01 (1788799381) [ 2640.192832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2640.224562] LNet: 137498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.236664] LNetError: 137498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2641.255762] LNet: Removed LNI 192.168.204.19@tcp2 [ 2641.273287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2641.402917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2641.428688] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2642.615690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2642.637472] LNet: 137640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.642547] LNetError: 137640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.657824] LNet: Removed LNI 192.168.204.19@tcp [ 2642.695666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2642.774221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2642.805110] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2649.324219] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:43:10 (1788799390) [ 2657.044507] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:43:18 (1788799398) [ 2663.018168] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:43:24 (1788799404) [ 2668.029857] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:43:29 (1788799409) [ 2673.086529] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:43:34 (1788799414) [ 2679.441789] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:43:40 (1788799420) [ 2685.576964] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:43:47 (1788799427) [ 2691.423692] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:43:52 (1788799432) [ 2697.203794] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:43:58 (1788799438) [ 2702.532817] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:44:03 (1788799443) [ 2708.108498] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:44:09 (1788799449) [ 2715.234700] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:44:16 (1788799456) [ 2723.107694] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2438 sec ========== 12:44:24 (1788799464) [ 2723.464854] LNet: 145105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.471603] LNetError: 145105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.481854] LNet: Removed LNI 192.168.204.19@tcp [ 2724.034779] Key type .llcrypt unregistered [ 2724.042326] Key type ._llcrypt unregistered [ 2735.347296] Key type ._llcrypt registered [ 2735.350478] Key type .llcrypt registered [ 2735.708085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2735.718493] alg: No test for adler32 (adler32-zlib) [ 2736.797646] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2737.014702] LNet: Added LNI 192.168.204.19@tcp [8/256/0/180] [ 2738.743278] Key type lgssc registered [ 2740.021110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.839664] Lustre: Mounted lustre-client - version 2.17.58_39_gc924a9e [ 2810.875381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2817.337715] Lustre: Unmounted lustre-client [ 2835.077408] Key type lgssc unregistered [ 2835.321635] LNet: 148049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.330200] LNetError: 148049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.350418] LNet: Removed LNI 192.168.204.19@tcp [ 2835.985163] Key type .llcrypt unregistered [ 2835.987521] Key type ._llcrypt unregistered