[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 556201668 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003459] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.009410] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012145] LSM: Security Framework initializing [ 0.013053] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.024010] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029618] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032009] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033104] *** VALIDATE tmpfs *** [ 0.034442] *** VALIDATE proc *** [ 0.037491] *** VALIDATE cgroup *** [ 0.039010] *** VALIDATE cgroup2 *** [ 0.041085] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043078] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045096] Spectre V2 : User space: Vulnerable [ 0.046008] Speculative Store Bypass: Vulnerable [ 0.048925] debug: unmapping init [mem 0xffffffff90a59000-0xffffffff90a60fff] [ 0.051251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053027] ... version: 2 [ 0.054016] ... bit width: 48 [ 0.054856] ... generic registers: 4 [ 0.055078] ... value mask: 0000ffffffffffff [ 0.056017] ... max period: 00007fffffffffff [ 0.057015] ... fixed-purpose events: 3 [ 0.058006] ... event mask: 000000070000000f [ 0.059265] rcu: Hierarchical SRCU implementation. [ 0.061290] smp: Bringing up secondary CPUs ... [ 0.062792] x86: Booting SMP configuration: [ 0.063076] .... node #0, CPUs: #1 #2 #3 [ 0.070111] smp: Brought up 1 node, 4 CPUs [ 0.072012] smpboot: Max logical packages: 1 [ 0.073010] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.134021] node 0 deferred pages initialised in 59ms [ 0.141138] devtmpfs: initialized [ 0.142308] x86/mm: Memory block size: 128MB [ 0.146171] gcov: version magic: 0x41383552 [ 0.149287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150180] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151260] pinctrl core: initialized pinctrl subsystem [ 0.152155] [ 0.152808] ************************************************************* [ 0.153012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154013] ** ** [ 0.155010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.157012] ** This means that this kernel is built to expose internal ** [ 0.158024] ** IOMMU data structures, which may compromise security on ** [ 0.159011] ** your system. ** [ 0.160011] ** ** [ 0.161011] ** If you see this message and you are not debugging the ** [ 0.162012] ** kernel, report this immediately to your vendor! ** [ 0.163008] ** ** [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ************************************************************* [ 0.167000] NET: Registered protocol family 16 [ 0.169560] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180166] cpuidle: using governor menu [ 0.181885] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184480] PCI: Using configuration type 1 for base access [ 0.187102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205079] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208517] cryptd: max_cpu_qlen set to 1000 [ 0.210349] ACPI: Added _OSI(Module Device) [ 0.211066] ACPI: Added _OSI(Processor Device) [ 0.213010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219701] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226331] ACPI: Interpreter enabled [ 0.228050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230014] ACPI: Using IOAPIC for interrupt routing [ 0.232140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251085] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260182] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266672] acpiphp: Slot [2] registered [ 0.269107] acpiphp: Slot [5] registered [ 0.270261] acpiphp: Slot [6] registered [ 0.272089] acpiphp: Slot [3] registered [ 0.273067] acpiphp: Slot [4] registered [ 0.274075] acpiphp: Slot [7] registered [ 0.276087] acpiphp: Slot [8] registered [ 0.277130] acpiphp: Slot [9] registered [ 0.279114] acpiphp: Slot [10] registered [ 0.281110] acpiphp: Slot [11] registered [ 0.283183] acpiphp: Slot [12] registered [ 0.285167] acpiphp: Slot [13] registered [ 0.287086] acpiphp: Slot [14] registered [ 0.288150] acpiphp: Slot [15] registered [ 0.290086] acpiphp: Slot [16] registered [ 0.292083] acpiphp: Slot [17] registered [ 0.294086] acpiphp: Slot [18] registered [ 0.295085] acpiphp: Slot [19] registered [ 0.297136] acpiphp: Slot [20] registered [ 0.299079] acpiphp: Slot [21] registered [ 0.300078] acpiphp: Slot [22] registered [ 0.302087] acpiphp: Slot [23] registered [ 0.304143] acpiphp: Slot [24] registered [ 0.306091] acpiphp: Slot [25] registered [ 0.307119] acpiphp: Slot [26] registered [ 0.309142] acpiphp: Slot [27] registered [ 0.310175] acpiphp: Slot [28] registered [ 0.312088] acpiphp: Slot [29] registered [ 0.314143] acpiphp: Slot [30] registered [ 0.316146] acpiphp: Slot [31] registered [ 0.318061] PCI host bridge to bus 0000:00 [ 0.320017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330062] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340235] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343572] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353022] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.358057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.372088] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376758] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385056] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.393936] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404742] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448010] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482068] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491613] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.494444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.496503] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.499432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.501211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.506149] iommu: Default domain type: Passthrough [ 0.508534] SCSI subsystem initialized [ 0.510127] ACPI: bus type USB registered [ 0.511150] usbcore: registered new interface driver usbfs [ 0.514095] usbcore: registered new interface driver hub [ 0.516066] usbcore: registered new device driver usb [ 0.517305] pps_core: LinuxPPS API ver. 1 registered [ 0.519017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.521109] PTP clock support registered [ 0.524075] EDAC MC: Ver: 3.0.0 [ 0.525710] PCI: Using ACPI for IRQ routing [ 0.526517] NetLabel: Initializing [ 0.528010] NetLabel: domain hash size = 128 [ 0.529010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.530093] NetLabel: unlabeled traffic allowed by default [ 0.533141] vgaarb: loaded [ 0.535028] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.536012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.544127] clocksource: Switched to clocksource kvm-clock [ 0.643559] VFS: Disk quotas dquot_6.6.0 [ 0.644595] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.646372] *** VALIDATE ramfs *** [ 0.647174] *** VALIDATE hugetlbfs *** [ 0.648194] pnp: PnP ACPI init [ 0.650118] pnp: PnP ACPI: found 6 devices [ 0.664655] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.667030] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.668609] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.670104] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.671984] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.674213] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.676753] NET: Registered protocol family 2 [ 0.678761] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.683829] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.686184] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.690679] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.693579] TCP: Hash tables configured (established 65536 bind 65536) [ 0.695908] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.698235] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700476] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.702758] NET: Registered protocol family 1 [ 0.704917] RPC: Registered named UNIX socket transport module. [ 0.707342] RPC: Registered udp transport module. [ 0.708639] RPC: Registered tcp transport module. [ 0.710480] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.713533] NET: Registered protocol family 44 [ 0.715373] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.717376] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.718970] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.721137] PCI: CLS 0 bytes, default 64 [ 0.723632] Unpacking initramfs... [ 2.361652] debug: unmapping init [mem 0xffff98443cc64000-0xffff98443ffcffff] [ 2.368600] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.371375] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.375688] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.945195] Initialise system trusted keyrings [ 2.946944] Key type blacklist registered [ 2.948835] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.959550] zbud: loaded [ 2.963465] *** VALIDATE nfs *** [ 2.965036] *** VALIDATE nfs4 *** [ 2.967892] pstore: using deflate compression [ 2.971849] Platform Keyring initialized [ 3.097466] NET: Registered protocol family 38 [ 3.099449] Key type asymmetric registered [ 3.101432] Asymmetric key parser 'x509' registered [ 3.103439] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.109175] io scheduler mq-deadline registered [ 3.111370] io scheduler kyber registered [ 3.113214] io scheduler bfq registered [ 3.115443] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.119212] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.122875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.126320] ACPI: Power Button [PWRF] [ 3.133154] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.141711] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.159555] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.191980] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.220331] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.225720] Non-volatile memory driver v1.3 [ 3.227371] Linux agpgart interface v0.103 [ 3.261977] virtio_blk virtio1: [vda] 139504 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.265021] vda: detected capacity change from 0 to 71426048 [ 3.289240] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.292291] vdb: detected capacity change from 0 to 1073741824 [ 3.335482] libphy: Fixed MDIO Bus: probed [ 3.413161] usbcore: registered new interface driver usbserial_generic [ 3.459202] usbserial: USB Serial support registered for generic [ 3.475153] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.516461] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.519929] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.524929] mousedev: PS/2 mouse device common for all mice [ 3.534469] rtc_cmos 00:05: RTC can wake from S4 [ 3.537603] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.543654] rtc_cmos 00:05: registered as rtc0 [ 3.634109] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.638339] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.700718] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.760183] intel_pstate: CPU model not supported [ 3.770361] hid: raw HID events driver (C) Jiri Kosina [ 3.772281] usbcore: registered new interface driver usbhid [ 3.774068] usbhid: USB HID core driver [ 3.775235] drop_monitor: Initializing network drop monitor service [ 3.777166] Initializing XFRM netlink socket [ 3.778928] NET: Registered protocol family 10 [ 3.781487] Segment Routing with IPv6 [ 3.782770] NET: Registered protocol family 17 [ 3.784496] mpls_gso: MPLS GSO support [ 3.789452] RAS: Correctable Errors collector initialized. [ 3.791123] AVX version of gcm_enc/dec engaged. [ 3.792398] AES CTR mode by8 optimization enabled [ 4.014803] sched_clock: Marking stable (4014710778, 0)->(5060628106, -1045917328) [ 4.018970] registered taskstats version 1 [ 4.021595] Loading compiled-in X.509 certificates [ 4.023950] zswap: loaded using pool lzo/zbud [ 4.059582] Key type big_key registered [ 4.075903] Key type encrypted registered [ 4.077732] ima: No TPM chip found, activating TPM-bypass! [ 4.079473] ima: Allocated hash algorithm: sha1 [ 4.081348] ima: No architecture policies found [ 4.083825] evm: Initialising EVM extended attributes: [ 4.086431] evm: security.selinux [ 4.087942] evm: security.ima [ 4.089214] evm: security.capability [ 4.090746] evm: HMAC attrs: 0x1 [ 4.093939] rtc_cmos 00:05: setting system clock to 2026-06-26 14:41:36 UTC (1782484896) [ 4.103370] debug: unmapping init [mem 0xffffffff91a03000-0xffffffff91bfffff] [ 4.107337] debug: unmapping init [mem 0xffffffff90782000-0xffffffff90a58fff] [ 4.117264] Write protecting the kernel read-only data: 28672k [ 4.120913] debug: unmapping init [mem 0xffffffff8ee03000-0xffffffff8effffff] [ 4.123930] debug: unmapping init [mem 0xffffffff8f714000-0xffffffff8f7fffff] [ 4.176515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.187516] systemd[1]: Detected virtualization kvm. [ 4.190472] systemd[1]: Detected architecture x86-64. [ 4.192982] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.227215] systemd[1]: No hostname configured. [ 4.229212] systemd[1]: Set hostname to . [ 4.231972] random: systemd: uninitialized urandom read (16 bytes read) [ 4.236701] systemd[1]: Initializing machine ID from random generator. [ 4.435560] random: systemd: uninitialized urandom read (16 bytes read) [ 4.438302] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.450318] random: systemd: uninitialized urandom read (16 bytes read) [ 4.453295] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.464285] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.490613] device-mapper: uevent: version 1.0.3 [ 5.493715] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.858523] virtio_net virtio0 ens2: renamed from eth0 [ 8.978267] scsi host0: ata_piix [ 9.408997] scsi host1: ata_piix [ 9.414410] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.418351] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.429829] random: crng init done [ 15.501093] random: 7 urandom warning(s) missed due to ratelimiting [ 18.427786] dracut-initqueue[576]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.446086] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.282518] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.646363] SELinux: Disabled at runtime. [ 25.927356] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.956312] systemd[1]: Detected virtualization kvm. [ 25.958151] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.013294] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.027150] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.038179] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.048536] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.056574] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.066598] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.081634] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 29.493833] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 31.971255] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.637042] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.887718] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.389285] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.690215] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 40.520296] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 42.401779] NFS: Registering the id_resolver key type [ 42.409809] Key type id_resolver registered [ 42.418564] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-client login: [ 65.361010] hrtimer: interrupt took 5000104 ns [ 119.751357] libcfs: loading out-of-tree module taints kernel. [ 119.889677] Key type ._llcrypt registered [ 119.896058] Key type .llcrypt registered [ 120.415149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 120.430694] alg: No test for adler32 (adler32-zlib) [ 121.798308] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 122.750830] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 124.511202] Key type lgssc registered [ 126.214058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 227.436024] Lustre: Mounted lustre-client [ 231.906495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.601421] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing check_logdir /tmp/testlogs/ [ 246.705417] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing yml_node [ 250.205535] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 252.293213] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 252.895336] Lustre: lustre-OST0000-osc-ffff984489020800: disconnect after 23s idle [ 254.933037] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 256.576384] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jun 26 10:45:47 EDT 2026 [ 269.003804] Lustre: Unmounted lustre-client [ 290.673944] Key type lgssc unregistered [ 290.874861] LNet: 8373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.882402] LNetError: 8373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.943677] LNet: Removed LNI 192.168.202.49@tcp [ 292.573355] Key type .llcrypt unregistered [ 292.575341] Key type ._llcrypt unregistered [ 311.690336] Key type ._llcrypt registered [ 311.692183] Key type .llcrypt registered [ 311.824812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 311.840665] alg: No test for adler32 (adler32-zlib) [ 312.705151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 312.746175] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 312.760144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 315.739819] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 316.032718] LNet: 9256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.041145] LNetError: 9256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.060286] LNet: Removed LNI 192.168.202.49@tcp [ 316.491387] Key type .llcrypt unregistered [ 316.492565] Key type ._llcrypt unregistered [ 320.210208] Key type ._llcrypt registered [ 320.212473] Key type .llcrypt registered [ 320.947369] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:46:52 (1782485212) [ 321.155212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 321.175088] alg: No test for adler32 (adler32-zlib) [ 322.053864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.098591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.112587] LNet: 9682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.130619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.168819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.186258] LNet: 9689:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 326.433185] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:46:58 (1782485218) [ 326.577128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.605339] LNet: 10265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.617466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.680913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 331.562347] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:47:02 (1782485222) [ 331.741288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.760722] LNet: 10850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.779897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.841542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 336.927838] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:47:08 (1782485228) [ 337.002605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.034103] LNet: 11427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.056244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.131286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 341.615434] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:47:13 (1782485233) [ 341.726516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.749447] LNet: 12020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.767277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.821751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 342.138300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 346.909918] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:47:18 (1782485238) [ 346.988861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.009138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.066238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 347.288785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 347.542282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 352.148522] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:47:23 (1782485243) [ 352.328633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.361554] LNet: 13205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.365637] LNet: 13205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 352.403142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.478481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 359.209544] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:47:30 (1782485250) [ 359.366135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.425700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.539903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 359.583379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.866930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.915206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.626922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 360.686796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.256193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 361.290209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.760842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 361.794070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 362.110946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 362.147915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.466655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 362.498908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.821684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 362.854316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.232342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 363.255818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.610500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 363.643924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.013392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 364.036850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.497373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 364.547298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.883370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 364.925096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 370.856765] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:47:41 (1782485261) [ 370.948220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.982328] LNet: 14468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.987598] LNet: 14468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 371.009384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.061782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 371.084983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 376.891907] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:47:48 (1782485268) [ 377.048611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.133328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.192705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 377.224864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 383.222594] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:47:54 (1782485274) [ 383.460402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.493549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.567290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 383.605450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 389.471682] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:48:00 (1782485280) [ 389.589857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.641955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.744888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 389.773964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 395.742621] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:48:06 (1782485286) [ 395.906694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.961651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.007540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 396.033217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 402.323038] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:48:13 (1782485293) [ 402.453699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.505517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.584496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 402.633254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.217254] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:48:19 (1782485299) [ 408.411133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.422189] LNet: 17991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.425176] LNet: 17991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 408.435001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.491454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.769506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.091188] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:48:25 (1782485305) [ 414.384952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.442884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.599279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.740468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 420.114507] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:48:31 (1782485311) [ 420.351411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.370822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.464098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.588875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.758324] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:48:37 (1782485317) [ 426.890540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.928407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.411491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.660793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.975787] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:48:45 (1782485325) [ 434.109596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.175495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.276923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.328890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.215482] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:48:51 (1782485331) [ 440.297121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.346918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.469638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.523372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.051792] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:48:58 (1782485338) [ 447.140553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.192703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.272300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 453.013318] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:49:04 (1782485344) [ 453.084878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.130882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.189509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 458.364378] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:49:09 (1782485349) [ 458.552307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.612971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.686849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 459.002230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 464.501370] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:49:15 (1782485355) [ 464.573908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.615795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.678340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 464.709244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 470.640055] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:49:21 (1782485361) [ 470.813368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.893679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.979185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 471.024534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 476.344730] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:49:27 (1782485367) [ 476.469092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.490222] LNet: 24597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.497414] LNet: 24597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 476.543968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.647323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 476.724746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 482.897401] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:49:34 (1782485374) [ 483.114628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.151723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.219439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 483.256396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 489.449982] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:49:40 (1782485380) [ 489.652969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.687468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.748910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 489.822745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 489.849195] LNetError: 25786:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 489.902938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 489.935962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 495.053411] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:49:46 (1782485386) [ 495.224033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.300079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.384400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8ny80RpmY5 [ 495.456947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8ny80RpmY5 [ 495.490286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8ny80RpmY5 [ 500.529990] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:49:51 (1782485391) [ 500.642262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.676295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.742359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 500.783891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 500.834256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 506.192108] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:49:57 (1782485397) [ 506.303881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.359591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.436652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 508.937583] LNet: Message forwarding will be enabled in 2s [ 512.031170] LNet: Message forwarding enabled [ 512.383616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 512.403526] LNet: Message forwarding will stop in 2s to 6s [ 516.890205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 517.331111] LNet: Message forwarding disabled [ 518.756726] LNet: Message forwarding will be enabled in 2s [ 522.463080] LNet: Message forwarding enabled [ 524.547522] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:50:15 (1782485415) [ 524.687867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.030547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.094256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 531.226251] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:50:22 (1782485422) [ 531.491751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.525151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.566233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 531.593151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 531.621284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 531.650515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 531.677733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 531.705985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 531.730844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 531.753864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 531.776907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 531.793886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 531.811701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 531.858244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 531.903983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 531.925624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 531.953926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 531.982107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 532.010193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 532.035328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 532.057776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 532.074909] LNetError: 28857:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 536.862533] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:50:28 (1782485428) [ 537.038420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.067494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.128887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.643472] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:50:34 (1782485434) [ 544.068664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg249-server [ 545.587890] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:50:36 (1782485436) [ 545.648696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.678132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.726673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.740432] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 547.001113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[50-54]@tcp [ 547.015444] LNet: 30572:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 547.045292] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 547.078631] LNetError: 30362:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.50@tcp has gone from up to down [ 548.086395] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 548.117943] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 549.107028] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.50@tcp is being used as a gateway but routing feature is not turned on [ 549.122291] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 551.144409] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 551.160297] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 552.380726] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:50:43 (1782485443) [ 554.249892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg249-server [ 555.246546] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 555.288150] LNetError: 30362:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 555.703683] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:50:46 (1782485446) [ 555.789507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.835487] LNetError: 31500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.859913] LNet: Removed LNI 192.168.202.49@tcp [ 555.890805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.969135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 556.019675] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 556.861893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[50-55/2]@tcp [ 556.897170] LNet: 31624:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 556.912178] LNet: 31624:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 556.936662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[50-55/2]@tcp [ 556.961095] LNetError: 31511:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.52@tcp has gone from up to down [ 556.974558] LNetError: 31511:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 562.512732] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:50:53 (1782485453) [ 562.635318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.662869] LNetError: 32212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.671065] LNet: Removed LNI 192.168.202.49@tcp [ 562.718163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.806853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 562.866491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.910129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.976961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.042393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.093949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.172138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 563.240948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.279996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.321346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.390495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.438098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.508327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 563.598530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.632056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.698296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.773491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.822276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.858561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 563.949525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.987528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.051244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 564.133252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.159806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.238306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 564.292039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.312454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.359258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 564.406687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.426372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.457692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 564.497481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.518311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.561909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.023245] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:51:01 (1782485461) [ 571.545878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg249-server [ 573.349122] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:51:04 (1782485464) [ 575.168221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg249-server [ 576.817265] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:51:07 (1782485467) [ 577.471178] Key type .llcrypt unregistered [ 577.473561] Key type ._llcrypt unregistered [ 581.987280] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 582.207487] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 582.210170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 582.216484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 584.802635] Key type ._llcrypt registered [ 584.804962] Key type .llcrypt registered [ 584.961419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.974729] alg: No test for adler32 (adler32-zlib) [ 585.867823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.914831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.933792] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 586.437552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 586.456547] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 587.704273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 587.717150] LNet: Removed LNI 10.1.2.3@tcp [ 587.816155] LNet: 34338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.832657] LNetError: 34338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.851697] LNet: Removed LNI 192.168.202.49@tcp [ 588.392509] Key type .llcrypt unregistered [ 588.394832] Key type ._llcrypt unregistered [ 598.330966] Key type ._llcrypt registered [ 598.333298] Key type .llcrypt registered [ 599.469795] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:51:30 (1782485490) [ 599.860557] Key type .llcrypt unregistered [ 599.864735] Key type ._llcrypt unregistered [ 605.384247] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 605.471129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 605.478941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 605.486072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 607.947827] Key type ._llcrypt registered [ 607.949562] Key type .llcrypt registered [ 608.063884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.073259] alg: No test for adler32 (adler32-zlib) [ 608.967023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.001161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.024134] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 609.529662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 609.563722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 611.460775] LNet: 35833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.470875] LNetError: 35833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.476721] LNet: Removed LNI 192.168.202.49@tcp [ 611.485793] LNet: Removed LNI 10.1.2.3@tcp [ 611.991429] Key type .llcrypt unregistered [ 611.992640] Key type ._llcrypt unregistered [ 620.655927] Key type ._llcrypt registered [ 620.658188] Key type .llcrypt registered [ 621.829871] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:51:52 (1782485512) [ 622.360553] Key type .llcrypt unregistered [ 622.362324] Key type ._llcrypt unregistered [ 627.341792] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 627.543461] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 627.548985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 627.555100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 630.139173] Key type ._llcrypt registered [ 630.142290] Key type .llcrypt registered [ 630.356274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 630.379739] alg: No test for adler32 (adler32-zlib) [ 631.296763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.363072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 631.403154] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 633.130085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 633.156507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 634.749665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 634.765912] LNet: Removed LNI 10.1.2.3@tcp [ 634.781702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 634.817695] LNet: Removed LNI 10.1.2.31@tcp [ 634.839286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 634.855143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 635.389545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 635.409542] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 636.202475] LNet: 37752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.211470] LNetError: 37752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 636.226239] LNet: Removed LNI 10.1.2.3@tcp [ 636.243349] LNet: Removed LNI 10.1.2.31@tcp [ 636.764627] Key type .llcrypt unregistered [ 636.766437] Key type ._llcrypt unregistered [ 645.791158] Key type ._llcrypt registered [ 645.792866] Key type .llcrypt registered [ 646.859456] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:52:18 (1782485538) [ 647.198054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.212296] alg: No test for adler32 (adler32-zlib) [ 648.129749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.195680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.235129] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 648.791141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.802089] LNet: 38704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.814850] LNetError: 38704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.826213] LNet: Removed LNI 192.168.202.49@tcp [ 648.851381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.883717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.49@tcp [ 648.900357] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 649.464342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.49@tcp [ 649.513709] LNet: Removed LNI 192.168.202.49@tcp [ 649.574627] LNet: 38840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.249620] Key type .llcrypt unregistered [ 650.255582] Key type ._llcrypt unregistered [ 659.588796] Key type ._llcrypt registered [ 659.590632] Key type .llcrypt registered [ 660.634844] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:52:31 (1782485551) [ 662.194467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg249-server [ 663.769155] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:52:35 (1782485555) [ 664.108473] Key type .llcrypt unregistered [ 664.113109] Key type ._llcrypt unregistered [ 669.126061] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 669.380303] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 669.392208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 669.404756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 671.993156] Key type ._llcrypt registered [ 672.000595] Key type .llcrypt registered [ 672.145037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.170129] alg: No test for adler32 (adler32-zlib) [ 673.043308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.120990] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 673.670722] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 675.142642] LNet: 40676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.146607] LNetError: 40676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.152573] LNet: Removed LNI 192.168.202.49@tcp1 [ 675.162165] LNet: Removed LNI 10.1.2.3@tcp2 [ 675.711518] Key type .llcrypt unregistered [ 675.713785] Key type ._llcrypt unregistered [ 685.090162] Key type ._llcrypt registered [ 685.092090] Key type .llcrypt registered [ 686.153912] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:52:57 (1782485577) [ 686.528195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.542259] alg: No test for adler32 (adler32-zlib) [ 687.467185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.538971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.556919] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 688.316639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.340277] LNet: 41644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.345936] LNetError: 41644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.358771] LNet: Removed LNI 192.168.202.49@tcp [ 688.376150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.435068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 688.457178] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 695.154289] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:53:06 (1782485586) [ 695.214839] LNet: 42226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.224510] LNetError: 42226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.251386] LNet: Removed LNI 192.168.202.49@tcp [ 695.872474] Key type .llcrypt unregistered [ 695.875240] Key type ._llcrypt unregistered [ 701.484904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.639885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 701.662397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 701.669545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.165768] Key type ._llcrypt registered [ 704.167199] Key type .llcrypt registered [ 704.387873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 704.403750] alg: No test for adler32 (adler32-zlib) [ 705.415098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.481714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.496845] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 706.040486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 706.063263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.807258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.822429] LNet: 42978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.828503] LNetError: 42978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.834596] LNet: Removed LNI 192.168.202.49@tcp [ 707.853779] LNet: Removed LNI 10.1.2.3@tcp [ 707.909453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.979558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 707.999058] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 708.003264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.294637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.307814] LNet: Removed LNI 192.168.202.49@tcp [ 708.316393] LNet: Removed LNI 10.1.2.3@tcp [ 708.326466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.408867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 708.441302] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 708.450334] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.993963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 709.022948] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 709.891299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.907368] LNet: 43167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.914740] LNet: 43167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 709.921445] LNetError: 43167:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.925440] LNetError: 43167:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 709.929851] LNet: Removed LNI 192.168.202.49@tcp [ 709.933815] LNet: Removed LNI 10.1.2.3@tcp [ 709.946350] LNet: Removed LNI 192.168.202.49@tcp2 [ 709.962193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.019171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 710.055610] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 710.059923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.064300] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 715.535382] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:53:26 (1782485606) [ 715.639135] LNet: 43756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.649608] LNetError: 43756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.662600] LNetError: 43756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 715.673749] LNet: Removed LNI 192.168.202.49@tcp [ 715.676351] LNet: Removed LNI 10.1.2.3@tcp [ 715.685571] LNet: Removed LNI 192.168.202.49@tcp2 [ 716.147644] Key type .llcrypt unregistered [ 716.149958] Key type ._llcrypt unregistered [ 720.807309] Key type ._llcrypt registered [ 720.808805] Key type .llcrypt registered [ 721.087370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.104936] alg: No test for adler32 (adler32-zlib) [ 721.955586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.005876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.026454] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 722.624035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.637357] LNet: 44159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.641658] LNetError: 44159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.657775] LNet: Removed LNI 192.168.202.49@tcp [ 722.679614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.727295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 722.748163] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 723.053754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.077929] LNet: Removed LNI 192.168.202.49@tcp [ 723.098615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.154310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 723.172093] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 723.553762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.573546] LNet: 44219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.581226] LNet: 44219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 723.594919] LNetError: 44219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.599845] LNetError: 44219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 723.635636] LNet: Removed LNI 192.168.202.49@tcp [ 723.656596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.725550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 723.743263] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 724.132887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.196040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.203783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.212911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.599188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.611832] LNet: 44298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.618022] LNetError: 44298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.637812] LNet: Removed LNI 192.168.202.49@tcp [ 726.649472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.710654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 726.732299] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 733.065732] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:53:43 (1782485623) [ 733.211918] LNet: 44885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.224704] LNetError: 44885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.252057] LNet: Removed LNI 192.168.202.49@tcp [ 733.877788] Key type .llcrypt unregistered [ 733.879899] Key type ._llcrypt unregistered [ 738.666814] Key type ._llcrypt registered [ 738.668956] Key type .llcrypt registered [ 738.833580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.852782] alg: No test for adler32 (adler32-zlib) [ 739.737143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.812795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.866218] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 740.597950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.618451] LNet: 45293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.622981] LNetError: 45293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.640864] LNet: Removed LNI 192.168.202.49@tcp [ 740.666496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.717618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 740.743243] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 746.260404] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:53:57 (1782485637) [ 746.453526] LNet: 45886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.459771] LNetError: 45886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.470525] LNet: Removed LNI 192.168.202.49@tcp [ 746.815583] Key type .llcrypt unregistered [ 746.817464] Key type ._llcrypt unregistered [ 752.752893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.918223] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.931422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.946229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.434684] Key type ._llcrypt registered [ 755.437208] Key type .llcrypt registered [ 755.614716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.633401] alg: No test for adler32 (adler32-zlib) [ 756.512708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.565564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.596539] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 757.186499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.205827] LNet: 46335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.211262] LNetError: 46335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.223729] LNet: Removed LNI 192.168.202.49@tcp [ 757.241980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.301832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 757.317141] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 762.201346] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:54:13 (1782485653) [ 762.284612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.314878] LNet: 46926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.322491] LNetError: 46926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.344173] LNet: Removed LNI 192.168.202.49@tcp [ 762.380473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.484250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 767.431586] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:54:18 (1782485658) [ 767.533483] LNet: 47509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.084784] Key type .llcrypt unregistered [ 768.086780] Key type ._llcrypt unregistered [ 773.328771] Key type ._llcrypt registered [ 773.338536] Key type .llcrypt registered [ 773.584676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.598671] alg: No test for adler32 (adler32-zlib) [ 774.517280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.572741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 774.600538] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 775.327563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.346754] LNet: 47913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.351282] LNetError: 47913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.361594] LNet: Removed LNI 192.168.202.49@tcp [ 775.378969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.440228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 775.463766] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 781.560620] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:54:32 (1782485672) [ 781.757950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.775765] LNet: 48506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.780476] LNetError: 48506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.801846] LNet: Removed LNI 192.168.202.49@tcp [ 781.832210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.929233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 788.812559] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:54:39 (1782485679) [ 790.771238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 793.706710] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:54:43 (1782485683) [ 798.134108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 801.437744] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:54:51 (1782485691) [ 804.446500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 807.226822] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:54:57 (1782485697) [ 809.195683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 811.678796] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:55:02 (1782485702) [ 814.273171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 816.780981] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:55:07 (1782485707) [ 817.102473] LNet: 50553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.190629] Key type .llcrypt unregistered [ 818.192943] Key type ._llcrypt unregistered [ 828.627366] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.886224] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.895290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.903787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 831.804424] Key type ._llcrypt registered [ 831.808773] Key type .llcrypt registered [ 832.438299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.509860] alg: No test for adler32 (adler32-zlib) [ 833.710366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.975098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 834.020051] LNet: Added LNI 192.168.202.49@tcp [8/512/0/180] [ 834.030956] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 834.037345] LNet: 50877:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 834.180101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.212212] LNet: 50896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.224959] LNetError: 50896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.232465] LNet: Removed LNI 192.168.202.49@tcp [ 834.235269] LNetError: 50896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 834.250387] LNet: Removed LNI 10.1.2.3@tcp [ 834.293880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.392698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 834.457236] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 834.641215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.722737] LNet: 50921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.745263] LNetError: 50921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.769918] LNet: Removed LNI 192.168.202.49@tcp [ 834.831401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.997945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 835.077122] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 835.089065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.098501] LNet: 50929:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 835.200979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.249546] LNet: 50947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.265664] LNetError: 50947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.277793] LNet: Removed LNI 192.168.202.49@tcp [ 835.287833] LNetError: 50947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 835.306354] LNet: Removed LNI 10.1.2.3@tcp [ 835.346566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.409469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 835.456786] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 835.466623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.474260] LNet: 50955:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 835.801446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.834384] LNet: 50981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.859257] LNetError: 50981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.869046] LNet: Removed LNI 192.168.202.49@tcp [ 835.874610] LNetError: 50981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 835.912776] LNet: Removed LNI 10.1.2.3@tcp [ 836.002934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.203909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 836.295182] LNet: Added LNI 192.168.202.49@tcp [8/256/0/182] [ 836.304961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 836.585722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.623562] LNet: 51009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.628336] LNetError: 51009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.636946] LNet: Removed LNI 192.168.202.49@tcp [ 836.641093] LNetError: 51009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.654552] LNet: Removed LNI 10.1.2.3@tcp [ 836.697969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.838292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 836.897115] LNet: Added LNI 192.168.202.49@tcp [8/256/0/182] [ 836.904862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 836.907607] LNet: 51017:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 837.060721] LNet: Removed LNI 192.168.202.49@tcp [ 837.120510] LNet: Removed LNI 10.1.2.3@tcp [ 838.815581] Key type .llcrypt unregistered [ 838.835375] Key type ._llcrypt unregistered [ 864.290801] Key type ._llcrypt registered [ 864.292152] Key type .llcrypt registered [ 865.440218] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:55:56 (1782485756) [ 866.138263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.187673] alg: No test for adler32 (adler32-zlib) [ 867.239029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.364234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.396532] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 868.255897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.49 [ 868.279367] LNet: 51996:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 868.296689] LNet: Removed LNI 192.168.202.49@tcp [ 876.352607] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:56:06 (1782485766) [ 876.637372] LNet: 52565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.643940] LNetError: 52565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.682961] LNet: Removed LNI 192.168.202.49@tcp [ 877.224478] Key type .llcrypt unregistered [ 877.230584] Key type ._llcrypt unregistered [ 886.274364] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 886.670411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 889.806810] Key type ._llcrypt registered [ 889.812282] Key type .llcrypt registered [ 890.229532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.250353] alg: No test for adler32 (adler32-zlib) [ 892.041341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.411971] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:56:30 (1782485790) [ 900.724966] LNet: 53483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.734939] LNetError: 53483:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.752830] LNet: Removed LNI 10.1.2.3@tcp [ 901.534553] Key type .llcrypt unregistered [ 901.545414] Key type ._llcrypt unregistered [ 909.634739] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.807587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.810978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.820926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 911.511182] Key type ._llcrypt registered [ 911.513994] Key type .llcrypt registered [ 911.670744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.684847] alg: No test for adler32 (adler32-zlib) [ 912.636486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.421414] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:56:50 (1782485810) [ 921.741608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 923.682258] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:56:54 (1782485814) [ 923.998144] LNet: 54661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.004311] LNetError: 54661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.014179] LNet: Removed LNI 10.1.2.3@tcp [ 924.935563] Key type .llcrypt unregistered [ 924.939134] Key type ._llcrypt unregistered [ 931.942363] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.312620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.316431] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.325990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.969484] Key type ._llcrypt registered [ 933.974232] Key type .llcrypt registered [ 934.184642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.209624] alg: No test for adler32 (adler32-zlib) [ 935.152886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.207477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 943.877222] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:57:15 (1782485835) [ 944.401846] LNet: 55883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.409770] LNetError: 55883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.427754] LNet: Removed LNI 10.1.2.3@tcp [ 945.030384] Key type .llcrypt unregistered [ 945.033893] Key type ._llcrypt unregistered [ 952.375426] Key type ._llcrypt registered [ 952.383242] Key type .llcrypt registered [ 952.721245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.750035] alg: No test for adler32 (adler32-zlib) [ 953.787559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 953.831073] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 962.797512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 963.544142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_interrupt -r 1 [ 963.600438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_interrupt -r 1 [ 963.651676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 963.675564] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 963.915335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 963.970148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 964.001234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 964.034381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 964.060936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 964.064366] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 964.100278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 964.140698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 964.188246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 964.219197] LNet: 56322:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 964.240659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 964.283368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 964.346105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 964.396449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 964.420457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 964.454778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 964.503404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 965.140817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_dropped -r 1 [ 965.186405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_dropped -r 1 [ 965.229662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 965.257264] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 965.261693] LNet: Skipped 1 previous similar message [ 965.507822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 965.539723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 965.574675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 965.613812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 965.665668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 965.686842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 965.768210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 965.832386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 965.876499] LNet: 56431:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 965.910147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 965.962603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 966.034122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 966.093800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 966.144315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 966.209826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 966.269033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 966.895599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_aborted -r 1 [ 966.943421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_aborted -r 1 [ 966.969722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 966.991135] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 967.198587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 967.234601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 967.260897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 967.287694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 967.340637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 967.374150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 967.408678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 967.438206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 967.463821] LNet: 56541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.492861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 967.534836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 967.571483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 967.610171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 967.646659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 967.682643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 967.728415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 968.349849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_no_route -r 1 [ 968.394573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_no_route -r 1 [ 968.435285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 968.743780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.779340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 968.806373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.832319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.864242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.907586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.948594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.988414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 969.055804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.101981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.150228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.187697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.224712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.272786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.332986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.020627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_timeout -r 1 [ 970.064779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_timeout -r 1 [ 970.107828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 970.141952] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 970.153398] LNet: Skipped 2 previous similar messages [ 970.509118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.558073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 970.618806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 970.704737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 970.758520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 970.829428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 970.884512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 970.936663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 970.976924] LNet: 56759:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.985379] LNet: 56759:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 971.010758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 971.069689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 971.106679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 971.146805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.180944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.230980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.265442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 971.824540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_error -r 1 [ 971.869853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_error -r 1 [ 971.920785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 972.165281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 972.189700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 972.218063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 972.251934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 972.276506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 972.305586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.334623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.383564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.437643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.466467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.509806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.555566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 972.580712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.614562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 972.650481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 972.946196] LNet: 56910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.958316] LNetError: 56910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.980159] LNet: Removed LNI 192.168.202.49@tcp [ 973.439975] Key type .llcrypt unregistered [ 973.442296] Key type ._llcrypt unregistered [ 993.761220] Key type ._llcrypt registered [ 993.762432] Key type .llcrypt registered [ 995.182508] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:58:06 (1782485886) [ 996.196433] Key type .llcrypt unregistered [ 996.205571] Key type ._llcrypt unregistered [ 1003.236432] Key type ._llcrypt registered [ 1003.238238] Key type .llcrypt registered [ 1003.545381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.572836] alg: No test for adler32 (adler32-zlib) [ 1004.615840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1004.666268] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1014.072603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1024.847719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.899871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1024.932248] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1026.452395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_interrupt -r 1 [ 1026.488361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_interrupt -r 1 [ 1026.518520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_interrupt -r 1 [ 1026.556486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_interrupt -r 1 [ 1026.594210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_interrupt -r 1 [ 1026.630909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_interrupt -r 1 [ 1026.680979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_interrupt -r 1 [ 1026.729868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_interrupt -r 1 [ 1026.771466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1026.789781] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1026.798451] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1027.072691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.101849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1027.136358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.170553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.206574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.245401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.279254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.351855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.379667] LNet: 58379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.406060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.462551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.505849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.546854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.617047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.651440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.679340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1028.121908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_dropped -r 1 [ 1028.150802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_dropped -r 1 [ 1028.184570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_dropped -r 1 [ 1028.218713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_dropped -r 1 [ 1028.249831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_dropped -r 1 [ 1028.280249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_dropped -r 1 [ 1028.307811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_dropped -r 1 [ 1028.335899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_dropped -r 1 [ 1028.396027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1028.431671] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1028.438158] LNet: Skipped 2 previous similar messages [ 1028.658883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.687669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.718754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.744948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.779889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.822287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.864213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.905482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.942615] LNet: 58501:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.995866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.081357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.115787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.184358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.211628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.237184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.263624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.852769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_aborted -r 1 [ 1029.889325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_aborted -r 1 [ 1029.925108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_aborted -r 1 [ 1029.957599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_aborted -r 1 [ 1029.989173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_aborted -r 1 [ 1030.028662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_aborted -r 1 [ 1030.059658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_aborted -r 1 [ 1030.091281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_aborted -r 1 [ 1030.132753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1030.163403] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1030.170448] LNet: Skipped 2 previous similar messages [ 1030.436463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.477581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.554099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.590549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.609356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.631962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.686150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.746680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.774376] LNet: 58622:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.791637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.821509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.857376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.887553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.912565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.942107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.977160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.515739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_no_route -r 1 [ 1031.566881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_no_route -r 1 [ 1031.624825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_no_route -r 1 [ 1031.664556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_no_route -r 1 [ 1031.716452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_no_route -r 1 [ 1031.760208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_no_route -r 1 [ 1031.805980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_no_route -r 1 [ 1031.876188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_no_route -r 1 [ 1031.938468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1032.202262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.240652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1032.263071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.285489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.308657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.330412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.353924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.373985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.397387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.421680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.451205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.473858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.501917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.542437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.573538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1033.069830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_timeout -r 1 [ 1033.121094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_timeout -r 1 [ 1033.191400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_timeout -r 1 [ 1033.254758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_timeout -r 1 [ 1033.318328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_timeout -r 1 [ 1033.380284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_timeout -r 1 [ 1033.443282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_timeout -r 1 [ 1033.488858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_timeout -r 1 [ 1033.552726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1033.581856] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1033.595200] LNet: Skipped 5 previous similar messages [ 1033.857123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.894781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.921196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.946580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.975763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.009539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.042762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.067968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.088751] LNet: 58865:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.093911] LNet: 58865:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1034.109948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.134633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.166752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.199189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.224637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.250817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.277859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.838104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_error -r 1 [ 1034.891501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_error -r 1 [ 1034.941879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_error -r 1 [ 1034.988279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_error -r 1 [ 1035.036803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_error -r 1 [ 1035.080969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_error -r 1 [ 1035.134275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_error -r 1 [ 1035.182918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_error -r 1 [ 1035.226633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1035.537873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.576662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.606831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.642835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.710333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.742666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.791880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.827673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.853321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.893586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.921040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.954872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.991641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.034662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.079496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1041.957654] LNet: 59062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.961987] LNetError: 59062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.983458] LNet: Removed LNI 192.168.202.49@tcp [ 1042.992460] LNetError: 59062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.021788] LNet: Removed LNI 192.168.202.49@tcp1 [ 1043.466674] Key type .llcrypt unregistered [ 1043.469185] Key type ._llcrypt unregistered [ 1061.259795] Key type ._llcrypt registered [ 1061.262466] Key type .llcrypt registered [ 1062.535104] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:59:13 (1782485953) [ 1063.557612] Key type .llcrypt unregistered [ 1063.562333] Key type ._llcrypt unregistered [ 1069.976473] Key type ._llcrypt registered [ 1069.978103] Key type .llcrypt registered [ 1070.188479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1070.216430] alg: No test for adler32 (adler32-zlib) [ 1071.230534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1071.274401] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1079.920741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1080.555973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_dropped -r 1 [ 1080.600354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_dropped -r 1 [ 1080.649837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1080.674531] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1080.894774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.921814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1080.954156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.986657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.020437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.046633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.078624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.103873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.121673] LNet: 60315:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.137974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.171969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.204914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.234765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.275345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.308739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.338788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.852928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_error -r 1 [ 1081.889293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_error -r 1 [ 1081.926079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1081.945629] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1082.157053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.182681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1082.205671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.231762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.279223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.320672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.373933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.408350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.427826] LNet: 60424:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.451139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.492729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.529790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.584659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.640556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.665810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.697852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1083.595524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_timeout -r 1 [ 1083.674189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_timeout -r 1 [ 1083.750879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1083.781299] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1084.134102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.163631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1084.217604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.256513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.291613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.322744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.347536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.367221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.380495] LNet: 60533:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.395114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.427613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.460470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.493441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.536472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.576755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.611566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.788972] LNet: 60575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.794225] LNetError: 60575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.814833] LNet: Removed LNI 192.168.202.49@tcp [ 1085.185648] Key type .llcrypt unregistered [ 1085.188467] Key type ._llcrypt unregistered [ 1103.835336] Key type ._llcrypt registered [ 1103.843318] Key type .llcrypt registered [ 1105.420960] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:59:56 (1782485996) [ 1106.254716] Key type .llcrypt unregistered [ 1106.256731] Key type ._llcrypt unregistered [ 1113.220703] Key type ._llcrypt registered [ 1113.223588] Key type .llcrypt registered [ 1113.424583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.443170] alg: No test for adler32 (adler32-zlib) [ 1114.405703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1114.441976] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1122.975726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1132.895448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.978937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1133.009355] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1134.928699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_dropped -r 1 [ 1134.987868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e remote_dropped -r 1 [ 1135.041268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_dropped -r 1 [ 1135.082606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e remote_dropped -r 1 [ 1135.143142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e remote_dropped -r 1 [ 1135.235254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e remote_dropped -r 1 [ 1135.322798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e remote_dropped -r 1 [ 1135.425597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e remote_dropped -r 1 [ 1135.532996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1135.570322] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1135.580940] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.149@tcp [ 1135.932081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.976863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1136.021501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.067969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.109945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.149268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.212830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.276373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.321286] LNet: 62044:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.357693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.407850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.467143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.508969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.549886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.579832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.612490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.645383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.154897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_error -r 1 [ 1137.184469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e remote_error -r 1 [ 1137.221902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_error -r 1 [ 1137.253334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e remote_error -r 1 [ 1137.284604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e remote_error -r 1 [ 1137.310484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e remote_error -r 1 [ 1137.339238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e remote_error -r 1 [ 1137.369807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e remote_error -r 1 [ 1137.400166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1137.415145] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1137.420163] LNet: Skipped 2 previous similar messages [ 1137.608292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.637622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1137.658550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.682265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.707309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.729875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.765249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.792832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.813993] LNet: 62167:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.832781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.861570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.903675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.933625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.960921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.993958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.058828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.089305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.776143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_timeout -r 1 [ 1138.811617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e remote_timeout -r 1 [ 1138.850691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_timeout -r 1 [ 1138.913872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e remote_timeout -r 1 [ 1138.974366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e remote_timeout -r 1 [ 1139.032274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e remote_timeout -r 1 [ 1139.085924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e remote_timeout -r 1 [ 1139.154184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e remote_timeout -r 1 [ 1139.186965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.149@tcp [ 1139.213620] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1139.535950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.571556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1139.616144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.674696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.701343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.729868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.775107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.808767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.830993] LNet: 62290:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.860090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.905602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.958542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.001651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.049439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.100314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.188693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.242518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.656605] LNet: 62368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.663668] LNetError: 62368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.671520] LNet: Removed LNI 192.168.202.49@tcp [ 1145.692657] LNet: Removed LNI 192.168.202.49@tcp1 [ 1146.403184] Key type .llcrypt unregistered [ 1146.405987] Key type ._llcrypt unregistered [ 1170.818194] Key type ._llcrypt registered [ 1170.823762] Key type .llcrypt registered [ 1172.643536] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:01:03 (1782486063) [ 1174.884868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1176.800266] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:01:07 (1782486067) [ 1178.187871] Key type .llcrypt unregistered [ 1178.192725] Key type ._llcrypt unregistered [ 1187.570674] Key type ._llcrypt registered [ 1187.572933] Key type .llcrypt registered [ 1187.967981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1187.997341] alg: No test for adler32 (adler32-zlib) [ 1188.939814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1188.964571] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1198.040874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1198.806502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e network_timeout -r 1 [ 1198.846684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e network_timeout -r 1 [ 1198.881479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1198.898834] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1199.133688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.172413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1199.201638] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1199.202662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.247903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.309351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.357916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.392495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.422391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.441866] LNet: 63931:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.459531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.487509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.519923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.572235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.653347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.731356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.153339] LNet: 63970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.162762] LNetError: 63970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.187901] LNet: Removed LNI 192.168.202.49@tcp [ 1200.798449] Key type .llcrypt unregistered [ 1200.802778] Key type ._llcrypt unregistered [ 1222.114913] Key type ._llcrypt registered [ 1222.116528] Key type .llcrypt registered [ 1222.319918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.333697] alg: No test for adler32 (adler32-zlib) [ 1223.257945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.287809] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1231.253241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1241.789389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.842915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.891593] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1243.846853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e network_timeout -r 1 [ 1243.887171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e network_timeout -r 1 [ 1243.927829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e network_timeout -r 1 [ 1243.967841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e network_timeout -r 1 [ 1244.057558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e network_timeout -r 1 [ 1244.114209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e network_timeout -r 1 [ 1244.180317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e network_timeout -r 1 [ 1244.235368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e network_timeout -r 1 [ 1244.318200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1244.382806] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1244.681672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.740146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1244.802412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.839125] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1244.842538] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.149@tcp [ 1244.851844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.907354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.948113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.987445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.034088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.054879] LNet: 64854:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.071952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.111970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.152877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.186481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.224475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.276589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.937991] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1251.315743] LNet: 64928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.330207] LNetError: 64928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.345816] LNet: Removed LNI 192.168.202.49@tcp [ 1251.390060] LNet: Removed LNI 192.168.202.49@tcp1 [ 1251.821768] Key type .llcrypt unregistered [ 1251.827992] Key type ._llcrypt unregistered [ 1268.279930] Key type ._llcrypt registered [ 1268.281795] Key type .llcrypt registered [ 1269.791709] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:02:40 (1782486160) [ 1270.204568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1270.228270] alg: No test for adler32 (adler32-zlib) [ 1271.173459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.267978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.304147] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1272.336244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1272.367950] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1273.260474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1273.347974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1273.397820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1273.433798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1273.484401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1273.540566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1273.598103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.49@tcp [ 1273.617537] LNetError: 66027:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.49@tcp to recovery [ 1274.271444] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1274.275523] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1276.320172] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1277.193074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.424776] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1280.997392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1281.041877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.135239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1281.181934] LNet: 66134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.193829] LNetError: 66134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.207444] LNet: Removed LNI 192.168.202.49@tcp [ 1281.225416] LNet: Removed LNI 192.168.202.49@tcp1 [ 1281.295788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.388244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1281.442240] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1282.639134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.671047] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1283.966828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1284.047940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.161273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1284.230954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1284.310956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1284.357366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1284.392222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1284.424961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.49@tcp [ 1284.444951] LNetError: 66407:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.49@tcp to recovery [ 1284.455266] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1286.938772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1290.593778] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1290.599759] LNet: Skipped 1 previous similar message [ 1294.932250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1294.973352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.015960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.719888] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:03:12 (1782486192) [ 1301.812940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.838572] LNet: 67107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.843965] LNetError: 67107:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.849587] LNetError: 67107:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1301.856631] LNet: Removed LNI 192.168.202.49@tcp [ 1301.869425] LNet: Removed LNI 192.168.202.49@tcp1 [ 1301.890961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.936320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.951489] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1302.549132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.584891] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1303.291823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1303.395942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1303.427889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1303.454364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1303.499954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1303.533980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1303.579944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.49@tcp --health 0 [ 1303.676445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1303.969628] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.49@tcp [ 1304.993534] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1305.009799] LNet: Skipped 1 previous similar message [ 1319.059118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1319.211750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.236277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1319.256467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.49@tcp --health 500 [ 1319.348865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.177740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.241447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.259172] LNet: 67605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.263946] LNetError: 67605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.269170] LNetError: 67605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1327.327173] LNet: Removed LNI 192.168.202.49@tcp [ 1327.336122] LNet: Removed LNI 192.168.202.49@tcp1 [ 1327.350769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.419915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1327.457259] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1328.254475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.279835] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1329.236510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1329.262969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.313949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1329.337385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1329.354509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1329.371639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1329.392069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1329.421586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.49@tcp --health 0 [ 1330.465473] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1330.468320] LNet: Skipped 3 previous similar messages [ 1330.693660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1341.796526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1341.848701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.877194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.649466] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:03:58 (1782486238) [ 1348.089459] LNet: 68663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.094514] LNetError: 68663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.099526] LNetError: 68663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1348.103979] LNet: Removed LNI 192.168.202.49@tcp [ 1348.114912] LNet: Removed LNI 192.168.202.49@tcp1 [ 1348.710418] Key type .llcrypt unregistered [ 1348.714257] Key type ._llcrypt unregistered [ 1354.131447] Key type ._llcrypt registered [ 1354.133465] Key type .llcrypt registered [ 1354.285361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.298762] alg: No test for adler32 (adler32-zlib) [ 1355.172584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.195713] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1363.372864] Lustre: DEBUG MARKER: Initial discovery [ 1363.421722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.149@tcp [ 1365.567165] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1365.691864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1365.737649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.149@tcp [ 1365.792303] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1365.825685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1365.872709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.149@tcp -r 1 -m GET -l 3 [ 1365.910589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.149@tcp -r 1 -m GET -e local_error [ 1365.944221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.149@tcp -r 1 -m PUT -l 6 [ 1366.495353] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1367.552494] Lustre: DEBUG MARKER: Force oleg249-client.virtnet to discover 192.168.202.149@tcp (in background) [ 1367.603321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.149@tcp [ 1368.876023] Lustre: DEBUG MARKER: Force oleg249-server to discover 192.168.202.49@tcp [ 1370.657316] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1370.821339] Lustre: DEBUG MARKER: Wait for 69416 [ 1372.324564] Lustre: DEBUG MARKER: Finished wait on 69416 [ 1372.557370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1372.606677] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1372.629979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1372.902122] LNet: 69875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.909481] LNetError: 69875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.939376] LNet: Removed LNI 192.168.202.49@tcp [ 1373.286787] Key type .llcrypt unregistered [ 1373.289735] Key type ._llcrypt unregistered [ 1387.956564] Key type ._llcrypt registered [ 1387.958613] Key type .llcrypt registered [ 1388.862824] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:04:40 (1782486280) [ 1389.410347] Key type .llcrypt unregistered [ 1389.418227] Key type ._llcrypt unregistered [ 1394.226220] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1394.370461] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1394.379461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1394.392135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1396.869922] Key type ._llcrypt registered [ 1396.873957] Key type .llcrypt registered [ 1397.032847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1397.043817] alg: No test for adler32 (adler32-zlib) [ 1397.932184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1397.974389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1397.990542] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1398.566898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1398.585063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1405.775578] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:04:57 (1782486297) [ 1406.081796] LNet: 71950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.089954] LNetError: 71950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.092664] LNet: Removed LNI 192.168.202.49@tcp [ 1406.110715] LNet: Removed LNI 10.1.2.3@tcp [ 1406.629497] Key type .llcrypt unregistered [ 1406.632096] Key type ._llcrypt unregistered [ 1412.469358] Key type ._llcrypt registered [ 1412.473460] Key type .llcrypt registered [ 1412.640857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.650454] alg: No test for adler32 (adler32-zlib) [ 1413.518789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.566458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.585768] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1414.214275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1414.233576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1415.036460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.49@tcp [ 1416.095335] LNetError: Unexpected error -22 connecting to 192.168.202.49@tcp at host 192.168.202.49:988 [ 1420.369890] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:05:11 (1782486311) [ 1420.570534] LNet: 73083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.579552] LNetError: 73083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.585068] LNet: Removed LNI 192.168.202.49@tcp [ 1420.593188] LNet: Removed LNI 10.1.2.3@tcp [ 1421.159430] Key type .llcrypt unregistered [ 1421.163406] Key type ._llcrypt unregistered [ 1426.930580] Key type ._llcrypt registered [ 1426.932581] Key type .llcrypt registered [ 1427.093955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1427.119358] alg: No test for adler32 (adler32-zlib) [ 1428.054433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.102758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1428.122750] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1428.683767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1428.707741] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1429.479363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.49@tcp1 --nid 192.168.202.49@tcp2 [ 1432.914190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp1 192.168.202.49@tcp2 [ 1432.924356] LNetError: 73917:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.49@tcp1 to 192.168.202.49@tcp2 - Route table may be misconfigured [ 1432.939754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp2 192.168.202.49@tcp1 [ 1438.362803] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:05:29 (1782486329) [ 1438.575191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.593844] LNet: 74490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.600568] LNetError: 74490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.647149] LNet: Removed LNI 192.168.202.49@tcp1 [ 1439.652733] LNetError: 74490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.665233] LNet: Removed LNI 192.168.202.49@tcp2 [ 1439.699250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.757980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.783521] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1440.373827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.391942] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1441.266891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1441.318482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.49@tcp [ 1441.356911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp1 192.168.202.49@tcp1 [ 1441.467780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.49@tcp -d 192.168.202.49@tcp -e network_timeout [ 1441.523662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -e network_timeout [ 1441.599736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -e network_timeout [ 1441.682754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -e network_timeout [ 1441.765747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1441.781995] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1441.798328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1441.893051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.011489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1442.784568] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1447.108489] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:05:38 (1782486338) [ 1447.206685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.241366] LNet: 75345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.250342] LNetError: 75345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.287183] LNet: Removed LNI 192.168.202.49@tcp [ 1449.317852] LNet: Removed LNI 192.168.202.49@tcp1 [ 1449.344511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.409532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1449.608231] LNet: 75389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.871944] Key type .llcrypt unregistered [ 1449.874258] Key type ._llcrypt unregistered [ 1458.102768] Key type ._llcrypt registered [ 1458.106446] Key type .llcrypt registered [ 1459.717285] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:05:50 (1782486350) [ 1460.129792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1460.140217] alg: No test for adler32 (adler32-zlib) [ 1461.108173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.378434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1464.111367] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:05:55 (1782486355) [ 1464.293183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.351361] LNet: 76485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.375958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.432791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.452599] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1465.112196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.124940] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1465.900567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1465.941798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp1 [ 1465.970382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp1 [ 1471.663788] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:06:02 (1782486362) [ 1473.224587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg249-server [ 1474.732223] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:06:05 (1782486365) [ 1476.302630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg249-server [ 1477.687830] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:06:09 (1782486369) [ 1479.238646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg249-server [ 1480.575025] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:06:11 (1782486371) [ 1481.963839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg249-server [ 1483.257778] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:06:14 (1782486374) [ 1484.558565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg249-server [ 1485.836990] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:06:17 (1782486377) [ 1487.223570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg249-server [ 1488.346421] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:06:19 (1782486379) [ 1489.643749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg249-server [ 1490.680215] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:06:22 (1782486382) [ 1491.870248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg249-server [ 1492.986608] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:06:24 (1782486384) [ 1494.150618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg249-server [ 1495.242477] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:06:26 (1782486386) [ 1495.420775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.436384] LNet: 80364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.441779] LNetError: 80364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.479195] LNet: Removed LNI 192.168.202.49@tcp [ 1496.480810] LNetError: 80364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.500980] LNet: Removed LNI 192.168.202.49@tcp1 [ 1496.521494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.564858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1496.583363] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1497.107444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1497.200241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1497.384803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.414497] LNet: 80511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.472915] LNet: Removed LNI 192.168.202.49@tcp [ 1498.500131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.565570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1498.592021] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1499.068748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1499.210769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.229449] LNetError: 80656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.234988] LNetError: 80656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1499.247904] LNet: Removed LNI 192.168.202.49@tcp [ 1499.272478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.305219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.326562] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1499.833557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1499.894323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1499.993130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.011538] LNet: 80803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.018928] LNet: 80803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1501.093688] LNet: Removed LNI 192.168.202.49@tcp [ 1501.108079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.154828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1501.178220] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1501.641553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1501.744673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.758029] LNetError: 80949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.761162] LNetError: 80949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.775736] LNet: Removed LNI 192.168.202.49@tcp [ 1501.796224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.841547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.860787] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1502.330921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1502.391239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1502.597145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.654037] LNet: Removed LNI 192.168.202.49@tcp [ 1503.676105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.727193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1503.747963] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1504.435708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1504.631352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.651484] LNet: 81241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.657354] LNet: 81241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1505.704156] LNet: Removed LNI 192.168.202.49@tcp [ 1505.720888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.780963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.798485] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1506.476560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1506.565367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1506.715290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.745173] LNetError: 81388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.749320] LNetError: 81388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1506.766377] LNet: Removed LNI 192.168.202.49@tcp [ 1506.783467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.832577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1506.853497] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1507.563714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1507.815764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.908604] LNet: Removed LNI 192.168.202.49@tcp [ 1508.933529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.025701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.064642] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1509.677263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1509.734847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1509.924124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.019887] LNet: Removed LNI 192.168.202.49@tcp [ 1511.048953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.126115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1511.169025] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1511.793913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1512.024242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.069492] LNet: Removed LNI 192.168.202.49@tcp [ 1513.111447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.180562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.198873] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1513.927584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1514.013753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1514.252684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.262722] LNet: 81973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.265841] LNet: 81973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1515.307188] LNet: Removed LNI 192.168.202.49@tcp [ 1515.331394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.398784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1515.421503] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1515.934779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1516.141684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.187768] LNetError: 82119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.194374] LNetError: 82119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1517.221902] LNet: Removed LNI 192.168.202.49@tcp [ 1517.242707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.303648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.317677] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1517.812518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1517.855696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.884920] LNet: Removed LNI 192.168.202.49@tcp [ 1517.898794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.942483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.960603] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1518.522636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1523.257924] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:06:54 (1782486414) [ 1523.322307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.358706] LNet: Removed LNI 192.168.202.49@tcp [ 1523.383453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.441285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1523.467608] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/180] [ 1524.381331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.410313] LNet: Removed LNI 192.168.202.49@tcp231 [ 1524.429902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.487497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1524.510262] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/180] [ 1524.514829] LNet: 83101:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.775774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1524.801634] LNet: Removed LNI 192.168.202.49@tcp231 [ 1524.819974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1524.844402] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/180] [ 1525.678360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.700908] LNet: Removed LNI 192.168.202.49@tcp231 [ 1525.718920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.787853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1525.819921] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/0] [ 1525.824951] LNet: 83254:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.122337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1526.165511] LNet: Removed LNI 192.168.202.49@tcp231 [ 1526.180068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1526.204763] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/0] [ 1527.097670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.131801] LNet: Removed LNI 192.168.202.49@tcp231 [ 1527.152798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.201937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1527.233835] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/60] [ 1527.259181] LNet: 83405:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1527.557859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1527.584908] LNet: Removed LNI 192.168.202.49@tcp231 [ 1527.598160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1527.615247] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/60] [ 1528.479759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.510342] LNet: Removed LNI 192.168.202.49@tcp231 [ 1528.526456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.582498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1528.607221] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/180] [ 1528.853655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1528.895958] LNet: Removed LNI 192.168.202.49@tcp231 [ 1528.918641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1528.952169] LNet: Added LNI 192.168.202.49@tcp231 [8/256/0/180] [ 1534.598889] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:07:06 (1782486426) [ 1534.697801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.725221] LNet: 84263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.731972] LNet: 84263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1534.740275] LNetError: 84263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.751096] LNetError: 84263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1534.775721] LNet: Removed LNI 192.168.202.49@tcp231 [ 1534.797191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.831283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1534.845222] LNet: Added LNI 192.168.202.49@tcp232 [8/256/0/180] [ 1535.369401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1535.401478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1535.430346] LNet: Removed LNI 192.168.202.49@tcp232 [ 1535.443670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1535.472852] LNet: Added LNI 192.168.202.49@tcp232 [8/256/0/180] [ 1536.134758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1536.169629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1536.200733] LNet: Removed LNI 192.168.202.49@tcp232 [ 1536.217903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1536.233465] LNet: Added LNI 192.168.202.49@tcp232 [8/256/0/180] [ 1536.737665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1536.768449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1536.799526] LNet: Removed LNI 192.168.202.49@tcp232 [ 1536.816307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1536.834791] LNet: Added LNI 192.168.202.49@tcp232 [8/256/0/180] [ 1537.351264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1537.382741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1537.406305] LNet: Removed LNI 192.168.202.49@tcp232 [ 1537.418061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1542.137934] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:07:13 (1782486433) [ 1543.006109] Key type .llcrypt unregistered [ 1543.007847] Key type ._llcrypt unregistered [ 1548.377190] Key type ._llcrypt registered [ 1548.378905] Key type .llcrypt registered [ 1548.525283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1548.539919] alg: No test for adler32 (adler32-zlib) [ 1549.449503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1549.470516] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1555.756061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1563.025686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.054869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1563.074091] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1564.141516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_interrupt -r 2 [ 1564.164483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_interrupt -r 2 [ 1564.189982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_interrupt -r 2 [ 1564.208946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_interrupt -r 2 [ 1564.230144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_interrupt -r 2 [ 1564.251613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_interrupt -r 2 [ 1564.283297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_interrupt -r 2 [ 1564.316593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_interrupt -r 2 [ 1564.351022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1564.369265] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1564.375972] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1564.561725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1564.581577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1564.608746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1564.632601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1564.655944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1564.673822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1564.699529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1564.735566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1564.753526] LNet: 86056:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1564.769845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1564.806146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1564.834594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1564.870284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1564.912104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1564.939817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1565.437303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_dropped -r 2 [ 1565.472963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_dropped -r 2 [ 1565.506896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_dropped -r 2 [ 1565.549985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_dropped -r 2 [ 1565.592632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_dropped -r 2 [ 1565.644448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_dropped -r 2 [ 1565.706657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_dropped -r 2 [ 1565.739189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_dropped -r 2 [ 1565.772372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1565.793953] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1566.092614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1566.133608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1566.163225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1566.185472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1566.207857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1566.229080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1566.254457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1566.276308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1566.290757] LNet: 86177:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1566.303417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1566.337940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1566.369993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1566.394523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1566.420510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1566.448139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1566.839958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_aborted -r 2 [ 1566.866596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_aborted -r 2 [ 1566.893901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_aborted -r 2 [ 1566.918495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_aborted -r 2 [ 1566.947970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_aborted -r 2 [ 1566.972829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_aborted -r 2 [ 1567.001207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_aborted -r 2 [ 1567.032683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_aborted -r 2 [ 1567.067638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1567.084428] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1567.283234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1567.329125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1567.366371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1567.408667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1567.474296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1567.502859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1567.547747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1567.599883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1567.627329] LNet: 86297:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.660747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1567.697189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.724588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1567.747517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.767988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1567.788611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1568.146447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_no_route -r 2 [ 1568.167836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_no_route -r 2 [ 1568.200371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_no_route -r 2 [ 1568.234901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_no_route -r 2 [ 1568.259457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_no_route -r 2 [ 1568.287215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_no_route -r 2 [ 1568.308541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_no_route -r 2 [ 1568.330739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_no_route -r 2 [ 1568.352895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1568.560252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1568.580771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1568.600332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1568.618365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1568.639610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1568.657483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1568.677873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1568.698851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1568.718429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1568.745532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1568.761994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1568.783089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1568.818243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1568.850494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1569.245240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e local_timeout -r 2 [ 1569.269264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e local_timeout -r 2 [ 1569.294424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e local_timeout -r 2 [ 1569.333272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e local_timeout -r 2 [ 1569.369724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e local_timeout -r 2 [ 1569.392134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e local_timeout -r 2 [ 1569.414665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e local_timeout -r 2 [ 1569.445153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e local_timeout -r 2 [ 1569.472957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1569.488232] LNet: There was an unexpected network error while writing to 192.168.202.149: rc = -22 [ 1569.492828] LNet: Skipped 1 previous similar message [ 1569.702836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1569.734323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1569.763393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1569.787764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1569.812781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1569.838206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1569.859307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1569.883446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1569.897926] LNet: 86537:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1569.902761] LNet: 86537:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1569.928423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1569.953606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1569.986321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1570.013525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1570.043549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1570.067120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1570.511740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp -m GET -e remote_dropped -r 2 [ 1570.532519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.149@tcp1 -m GET -e remote_dropped -r 2 [ 1570.563164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp -m GET -e remote_dropped -r 2 [ 1570.584983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp -d 192.168.202.49@tcp1 -m GET -e remote_dropped -r 2 [ 1570.608938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp -m GET -e remote_dropped -r 2 [ 1570.630377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.49@tcp1 -d 192.168.202.149@tcp1 -m GET -e remote_dropped -r 2 [ 1570.658246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp -m GET -e remote_dropped -r 2 [ 1570.693896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.49@tcp1 -d 192.168.202.49@tcp1 -m GET -e remote_dropped -r 2 [ 1570.724211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.49@tcp 192.168.202.149@tcp [ 1570.741765] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.149@tcp [ 1570.896247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1570.913799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1570.936885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1570.961590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1570.987398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1571.013846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1571.039771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1571.064756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1571.087844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1571.114973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1571.132891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1571.154328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1571.179362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1571.202479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1575.402882] LNet: 86731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.407878] LNetError: 86731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.413764] LNet: Removed LNI 192.168.202.49@tcp [ 1575.422660] LNet: Removed LNI 192.168.202.49@tcp1 [ 1575.762308] Key type .llcrypt unregistered [ 1575.766184] Key type ._llcrypt unregistered [ 1588.235788] Key type ._llcrypt registered [ 1588.238318] Key type .llcrypt registered [ 1589.205552] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:08:00 (1782486480) [ 1589.659954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.701378] alg: No test for adler32 (adler32-zlib) [ 1590.613286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.669141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.715674] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1591.462116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1591.483786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.49@tcp [ 1591.494641] LNetError: 87709:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.49@tcp to recovery [ 1591.570908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.49@tcp [ 1591.639148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1595.926884] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:08:07 (1782486487) [ 1596.085428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.128254] LNet: 88298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.139382] LNetError: 88298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.153870] LNet: Removed LNI 192.168.202.49@tcp [ 1596.168313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.204623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.231845] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1596.776229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1596.792938] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1597.496062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.49@tcp [ 1597.510378] LNetError: 88564:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.49@tcp to recovery [ 1598.239358] LNet: 1 local NIs in recovery (showing 1): 192.168.202.49@tcp [ 1598.240651] LNet: There was an unexpected network error while writing to 192.168.202.49: rc = -22 [ 1598.715674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.49@tcp [ 1605.930286] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:08:17 (1782486497) [ 1607.141504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg249-server [ 1608.470182] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:08:19 (1782486499) [ 1609.977579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg249-server [ 1611.121215] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:08:22 (1782486502) [ 1612.459210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg249-server [ 1613.882576] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:08:25 (1782486505) [ 1613.961747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.982746] LNet: 90187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.986705] LNetError: 90187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.993709] LNet: Removed LNI 192.168.202.49@tcp [ 1614.002865] LNet: Removed LNI 192.168.202.49@tcp1 [ 1614.022204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.089106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.108673] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1614.591318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1614.634705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1614.662423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1614.711463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1614.748176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1614.780360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1619.575101] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:08:31 (1782486511) [ 1619.639664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.656647] LNet: 90915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.661534] LNetError: 90915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.666300] LNetError: 90915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1619.680840] LNet: Removed LNI 192.168.202.49@tcp [ 1619.699544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.744552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.777402] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1624.577410] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:08:36 (1782486516) [ 1624.821193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.837205] LNet: 91622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.840505] LNetError: 91622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.852962] LNet: Removed LNI 192.168.202.49@tcp [ 1624.887671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.944256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.971794] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1630.316517] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:08:41 (1782486521) [ 1631.424980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1632.943371] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:08:44 (1782486524) [ 1633.321136] LNet: 92656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.325321] LNetError: 92656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.336149] LNet: Removed LNI 192.168.202.49@tcp [ 1633.755929] Key type .llcrypt unregistered [ 1633.761612] Key type ._llcrypt unregistered [ 1637.819708] Key type ._llcrypt registered [ 1637.822522] Key type .llcrypt registered [ 1637.982828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1637.993736] alg: No test for adler32 (adler32-zlib) [ 1638.863254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1638.887092] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1644.446665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1647.908607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.149@tcp [ 1648.195284] LNet: 93073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.199842] LNetError: 93073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.228210] LNet: Removed LNI 192.168.202.49@tcp [ 1648.548313] Key type .llcrypt unregistered [ 1648.551114] Key type ._llcrypt unregistered [ 1657.704189] Key type ._llcrypt registered [ 1657.707177] Key type .llcrypt registered [ 1658.910983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1659.912158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1661.163695] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:09:12 (1782486552) [ 1662.204465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1663.432744] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:09:14 (1782486554) [ 1664.683527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg249-server [ 1666.046929] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:09:17 (1782486557) [ 1667.314164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg249-server [ 1668.534148] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:09:19 (1782486559) [ 1669.637104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.674479] alg: No test for adler32 (adler32-zlib) [ 1670.735971] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1670.979732] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1672.681052] Key type lgssc registered [ 1673.663813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1687.378754] Key type lgssc unregistered [ 1687.573485] LNet: 96246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.578914] LNetError: 96246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.596801] LNet: Removed LNI 192.168.202.49@tcp [ 1688.028560] Key type .llcrypt unregistered [ 1688.029841] Key type ._llcrypt unregistered [ 1698.308161] Key type ._llcrypt registered [ 1698.310583] Key type .llcrypt registered [ 1699.157370] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:09:50 (1782486590) [ 1699.438663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.456214] alg: No test for adler32 (adler32-zlib) [ 1700.319829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.368255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1700.400406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1700.447845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1700.478566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1705.592608] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:09:56 (1782486596) [ 1705.672366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.695467] LNet: 97656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.716760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.786647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.813396] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1706.659549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.675577] LNet: 97804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.681189] LNetError: 97804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.693602] LNet: Removed LNI 192.168.202.49@tcp [ 1706.717839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.796326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1706.815633] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1707.281978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.294640] LNetError: 97832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.310489] LNet: Removed LNI 192.168.202.49@tcp [ 1707.325434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.379798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1707.406482] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1712.582907] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:10:03 (1782486603) [ 1712.670994] LNet: 98414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.680347] LNet: 98414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1712.694121] LNetError: 98414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.725098] LNet: Removed LNI 192.168.202.49@tcp [ 1713.231524] Key type .llcrypt unregistered [ 1713.236227] Key type ._llcrypt unregistered [ 1717.707101] Key type ._llcrypt registered [ 1717.710446] Key type .llcrypt registered [ 1717.887914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1717.899547] alg: No test for adler32 (adler32-zlib) [ 1718.787106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.855984] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1718.862657] LNet: *** cfs_fail_loc=e003, val=100*** [ 1719.455640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1719.470797] LNet: *** cfs_fail_loc=e003, val=100*** [ 1719.472600] LNet: Skipped 2 previous similar messages [ 1719.478447] Ping reply received with mapping: [ 1719.479887] found metadata reply NI[192.168.202.49@tcp] [ 1719.481699] 31 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1719.484632] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1719.487584] 5a 5a 5a 5a 5a 5a 5a 5a [ 1719.563992] LNet: 98819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.571986] LNetError: 98819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.589619] LNet: Removed LNI 192.168.202.49@tcp [ 1720.018625] Key type .llcrypt unregistered [ 1720.020082] Key type ._llcrypt unregistered [ 1727.805703] Key type ._llcrypt registered [ 1727.808267] Key type .llcrypt registered [ 1728.825809] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:10:20 (1782486620) [ 1729.121433] Key type .llcrypt unregistered [ 1729.122666] Key type ._llcrypt unregistered [ 1733.151160] Key type ._llcrypt registered [ 1733.152384] Key type .llcrypt registered [ 1733.332736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.347889] alg: No test for adler32 (adler32-zlib) [ 1734.352463] LNetError: 99865:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1734.613684] Key type .llcrypt unregistered [ 1734.615446] Key type ._llcrypt unregistered [ 1738.042395] Key type ._llcrypt registered [ 1738.043644] Key type .llcrypt registered [ 1738.940206] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:10:30 (1782486630) [ 1739.301305] Key type .llcrypt unregistered [ 1739.303302] Key type ._llcrypt unregistered [ 1743.625780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1743.632972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1746.083782] Key type ._llcrypt registered [ 1746.085933] Key type .llcrypt registered [ 1746.240378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1746.255848] alg: No test for adler32 (adler32-zlib) [ 1747.124907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.167060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1753.511470] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:10:45 (1782486645) [ 1753.594590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.618404] LNet: 101639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.627216] LNetError: 101639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1753.638170] LNet: Removed LNI 10.1.2.3@tcp [ 1753.651889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.194635] Key type .llcrypt unregistered [ 1754.196710] Key type ._llcrypt unregistered [ 1760.392426] Key type ._llcrypt registered [ 1760.410156] Key type .llcrypt registered [ 1760.675027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1760.688993] alg: No test for adler32 (adler32-zlib) [ 1761.706834] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1761.751668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1761.866818] LNet: 101934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.871667] LNetError: 101934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.922419] LNet: Removed LNI 192.168.202.49@tcp [ 1763.568486] Key type .llcrypt unregistered [ 1763.569883] Key type ._llcrypt unregistered [ 1776.513939] Key type ._llcrypt registered [ 1776.516304] Key type .llcrypt registered [ 1778.041543] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:11:08 (1782486668) [ 1778.799418] Key type .llcrypt unregistered [ 1778.805351] Key type ._llcrypt unregistered [ 1785.064737] Key type ._llcrypt registered [ 1785.068705] Key type .llcrypt registered [ 1785.412326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.429853] alg: No test for adler32 (adler32-zlib) [ 1793.178344] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:11:24 (1782486684) [ 1793.628631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.685876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.817022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.922841] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1795.394371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.449769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1795.514390] LNet: Removed LNI 192.168.202.49@tcp [ 1795.938946] LNet: 103679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.587611] Key type .llcrypt unregistered [ 1796.589212] Key type ._llcrypt unregistered [ 1813.832866] Key type ._llcrypt registered [ 1813.834353] Key type .llcrypt registered [ 1816.140552] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:11:45 (1782486705) [ 1816.910829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.972344] alg: No test for adler32 (adler32-zlib) [ 1818.084905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.191909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.233080] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1819.029172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1819.097616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.49@tcp [ 1819.125974] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1819.131554] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1819.136028] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1819, 0, 0 [ 1819.144564] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1819.151610] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1819.167362] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1819.183075] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1819, 0, 0 [ 1819.195623] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1819.197340] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1819.204085] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1819.206233] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1819, 0, 1869 [ 1819.210901] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1819.216221] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1819.222964] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1819.238112] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1819, 0, 1869 [ 1819.246877] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1827.252711] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:11:57 (1782486717) [ 1827.989799] LNet: 105222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.999538] LNetError: 105222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.018043] LNet: Removed LNI 192.168.202.49@tcp [ 1829.014515] Key type .llcrypt unregistered [ 1829.020137] Key type ._llcrypt unregistered [ 1836.853427] Key type ._llcrypt registered [ 1836.859150] Key type .llcrypt registered [ 1837.056114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1837.071318] alg: No test for adler32 (adler32-zlib) [ 1837.974958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1838.014111] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1845.188784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 1854.305885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.360949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1854.404739] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 1855.479941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1855.617809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1855.621670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1855.625039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1857.964507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1858.002941] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1868.058415] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:12:39 (1782486759) [ 1868.186920] LNet: 106729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.196747] LNetError: 106729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.211825] LNet: Removed LNI 192.168.202.49@tcp [ 1869.279259] LNet: Removed LNI 192.168.202.49@tcp1 [ 1869.282359] LNetError: 106729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1869.291304] LNetError: 106729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1869.337504] LNet: Removed LNI 10.1.2.3@tcp99 [ 1870.093579] Key type .llcrypt unregistered [ 1870.095321] Key type ._llcrypt unregistered [ 1876.476268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1876.479900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1878.908687] Key type ._llcrypt registered [ 1878.911191] Key type .llcrypt registered [ 1879.069852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1879.080540] alg: No test for adler32 (adler32-zlib) [ 1880.014709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.066954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.104835] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1880.609458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1880.630538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1882.179281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1882.225256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1882.262819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1882.290454] LNetError: 107476:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1882.322957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1882.363993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1882.763051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1882.851292] LNet: 107496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.860075] LNetError: 107496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.865711] LNet: Removed LNI 192.168.202.49@tcp [ 1882.879783] LNet: Removed LNI 10.1.2.3@tcp [ 1883.374346] Key type .llcrypt unregistered [ 1883.376031] Key type ._llcrypt unregistered [ 1894.616561] Key type ._llcrypt registered [ 1894.618286] Key type .llcrypt registered [ 1895.980166] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:13:06 (1782486786) [ 1896.401457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.439233] alg: No test for adler32 (adler32-zlib) [ 1897.362890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.401839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1897.417936] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1898.079229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.49@tcp [ 1898.168675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg249-client@tcp [ 1903.436239] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:13:14 (1782486794) [ 1903.525548] LNet: 109021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.537350] LNetError: 109021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.561382] LNet: Removed LNI 192.168.202.49@tcp [ 1904.117467] Key type .llcrypt unregistered [ 1904.124542] Key type ._llcrypt unregistered [ 1909.077948] Key type ._llcrypt registered [ 1909.080441] Key type .llcrypt registered [ 1909.291449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.313498] alg: No test for adler32 (adler32-zlib) [ 1910.188536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1910.206243] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1910.240876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1915.770527] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:13:27 (1782486807) [ 1917.027188] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1918.642408] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:13:29 (1782486809) [ 1919.785231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1921.165913] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:13:32 (1782486812) [ 1921.266081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.280311] LNet: 110444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.293113] LNetError: 110444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.306948] LNet: Removed LNI 192.168.202.49@tcp [ 1921.340754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.373659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.390411] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1922.076822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1922.122649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1928.157410] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:13:39 (1782486819) [ 1928.263672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.294587] LNet: 111153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.312424] LNetError: 111153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.331977] LNet: Removed LNI 192.168.202.49@tcp [ 1928.364408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.429773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1928.454305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1928.949305] Key type .llcrypt unregistered [ 1928.951757] Key type ._llcrypt unregistered [ 1938.524440] Key type ._llcrypt registered [ 1938.526365] Key type .llcrypt registered [ 1939.642359] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:13:50 (1782486830) [ 1940.012282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.032857] alg: No test for adler32 (adler32-zlib) [ 1940.957762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.052689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.083922] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1941.795962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1941.890140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1948.543362] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:13:59 (1782486839) [ 1948.679961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.697877] LNet: 112709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.707627] LNetError: 112709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.735396] LNet: Removed LNI 192.168.202.49@tcp [ 1949.768932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.838656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1949.860243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1955.328673] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:14:06 (1782486846) [ 1955.581526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.603749] LNet: 113294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.628598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.703230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1956.015772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1956.036572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1956.080178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1956.453629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1956.512328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1956.601630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.646641] LNet: 113327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.669450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.759453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1957.048163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1957.078869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1957.144235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1957.665459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1957.704576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1963.603712] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:14:14 (1782486854) [ 1963.732142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.764245] LNet: 113925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.806272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.890614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.918806] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1964.626672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1964.649200] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1965.600888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1965.680787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1965.952386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.980143] LNetError: 114207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.007214] LNet: Removed LNI 192.168.202.49@tcp [ 1967.010512] LNetError: 114207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.037629] LNet: Removed LNI 192.168.202.49@tcp2 [ 1967.061742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.136356] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1967.142390] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1967.146762] LNet: 114214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1967.752791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1968.002918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.018584] LNet: 114251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.023365] LNet: 114251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1969.055211] LNet: Removed LNI 192.168.202.49@tcp [ 1969.063418] LNetError: 114251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.072933] LNetError: 114251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1969.090626] LNet: Removed LNI 192.168.202.49@tcp2 [ 1969.117889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.199645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.229849] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1970.018344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1970.036509] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1970.750533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1970.782528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1970.830376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1971.028512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.065371] LNet: Removed LNI 192.168.202.49@tcp [ 1973.093565] LNet: Removed LNI 192.168.202.49@tcp2 [ 1973.127822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.177559] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1973.182770] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1973.186692] LNet: 114541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1973.724505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1973.928443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.947417] LNetError: 114579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.952798] LNetError: 114579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1973.959460] LNet: Removed LNI 192.168.202.49@tcp [ 1975.016435] LNet: Removed LNI 192.168.202.49@tcp2 [ 1975.064488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.132411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.154564] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1975.852636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1975.871119] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1976.526866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1976.558595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1976.722716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.738243] LNet: 114858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.742540] LNet: 114858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1977.759217] LNet: Removed LNI 192.168.202.49@tcp [ 1977.780593] LNet: Removed LNI 192.168.202.49@tcp2 [ 1977.794965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.835760] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1977.839595] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1977.846861] LNet: 114864:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1978.265698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1978.470239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1979.488545] LNet: Removed LNI 192.168.202.49@tcp [ 1979.503479] LNet: Removed LNI 192.168.202.49@tcp2 [ 1979.546760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.623594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.654511] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1980.245323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1980.263444] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1981.037756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1981.079201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1981.126285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1981.355978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.432604] LNet: Removed LNI 192.168.202.49@tcp [ 1982.438602] LNetError: 115186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.457174] LNetError: 115186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1982.473272] LNet: Removed LNI 192.168.202.49@tcp2 [ 1982.501516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.585460] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1982.590618] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1982.594492] LNet: 115192:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1983.046213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1983.225957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.251378] LNet: Removed LNI 192.168.202.49@tcp [ 1984.299546] LNet: Removed LNI 192.168.202.49@tcp2 [ 1984.343251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.464118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1984.503380] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1985.112868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1985.153447] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1986.029448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1986.059493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1986.127333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1986.493515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.553283] LNet: Removed LNI 192.168.202.49@tcp [ 1987.571398] LNet: Removed LNI 192.168.202.49@tcp2 [ 1987.602959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.703806] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1987.712599] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1987.715382] LNet: 115531:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1988.170869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1988.423475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.471332] LNet: Removed LNI 192.168.202.49@tcp [ 1989.480975] LNet: Removed LNI 192.168.202.49@tcp2 [ 1989.531495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.599798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.631426] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1990.435412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1990.455707] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1991.288315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1991.318734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1991.347209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1991.380542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 1991.663877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.735157] LNet: Removed LNI 192.168.202.49@tcp [ 1993.787272] LNet: Removed LNI 192.168.202.49@tcp2 [ 1993.876942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.964561] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 1993.975637] LNet: Added LNI 192.168.202.49@tcp2 [8/256/0/180] [ 1994.693230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 2001.256561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2002.879836] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:14:53 (1782486893) [ 2002.995727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.018864] LNet: 116645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.027564] LNet: 116645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2003.035777] LNetError: 116645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.041571] LNetError: 116645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 2003.046504] LNet: Removed LNI 192.168.202.49@tcp [ 2004.074084] LNet: Removed LNI 192.168.202.49@tcp2 [ 2004.105772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.416348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.437094] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2005.105395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2005.147337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2005.898324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.918746] LNet: Removed LNI 192.168.202.49@tcp [ 2005.928694] LNet: Removed LNI 10.1.2.3@tcp [ 2005.946324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.034734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2006.076970] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2006.084541] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2006.562642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2006.599438] LNet: Removed LNI 192.168.202.49@tcp [ 2006.617272] LNet: Removed LNI 10.1.2.3@tcp [ 2012.741757] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:15:03 (1782486903) [ 2012.940675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.998264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.106815] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2013.212645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2013.218692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2013.225588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2015.634916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.668994] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2016.247881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2016.262676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2017.804551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.49@tcp [ 2017.840545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2017.886777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.49@tcp -m PUT -e local_timeout -r 1 [ 2017.968436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2018.024694] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2018.027230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2018.237240] LNetError: 117590:0:(lib-move.c:4714:lnet_parse()) 192.168.202.49@tcp, src 192.168.202.49@tcp: Dropping PUT (error -108 looking up sender) [ 2020.321222] LNet: Removed LNI 192.168.202.49@tcp [ 2020.330644] LNet: Removed LNI 10.1.2.3@tcp [ 2020.757351] Key type .llcrypt unregistered [ 2020.759538] Key type ._llcrypt unregistered [ 2030.399528] Key type ._llcrypt registered [ 2030.401511] Key type .llcrypt registered [ 2031.425469] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:15:22 (1782486922) [ 2031.757851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.772781] alg: No test for adler32 (adler32-zlib) [ 2032.831246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.513212] LNet: 118901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.216414] Key type .llcrypt unregistered [ 2034.217970] Key type ._llcrypt unregistered [ 2039.876755] Key type ._llcrypt registered [ 2039.883697] Key type .llcrypt registered [ 2040.089284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.110333] alg: No test for adler32 (adler32-zlib) [ 2040.992345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2041.010372] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2048.177551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 2048.351840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2048.808759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.149@tcp [ 2049.006434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2049.025893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2049.040549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2049.072638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2049.105310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2049.136981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2049.162510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2049.184340] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.149@tcp [ 2049.201589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2049.219919] LNet: 119350:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2049.238534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2049.281861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2049.324944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2049.377455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2049.425682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2049.472537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2050.105831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2050.292758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2050.314069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2050.338564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2050.367396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2050.408993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2050.433929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2050.478714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2050.524692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2050.544729] LNet: 119467:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2050.577890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2050.643906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2050.684302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2050.719193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2050.750982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2050.806774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2051.609367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.149@tcp [ 2052.039228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2052.087349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2052.125778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2052.167954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2052.204821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2052.226728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2052.251415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2052.287924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2052.300826] LNet: 119585:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2052.317573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2052.345792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2052.377830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2052.406572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2052.437091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2052.472792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2053.363739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2053.664495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2053.719957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2053.757637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2053.793854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2053.829288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2053.856457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2053.891706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2053.919715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2053.973203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2054.024479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2054.091040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2054.121682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2054.172170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2054.205480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2054.560545] LNet: 119748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.565615] LNetError: 119748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.582996] LNet: Removed LNI 192.168.202.49@tcp [ 2055.027179] Key type .llcrypt unregistered [ 2055.029629] Key type ._llcrypt unregistered [ 2069.102260] Key type ._llcrypt registered [ 2069.104278] Key type .llcrypt registered [ 2070.289113] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:16:01 (1782486961) [ 2070.713298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2070.732549] alg: No test for adler32 (adler32-zlib) [ 2071.635544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.929294] LNet: 120634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.397370] Key type .llcrypt unregistered [ 2072.399608] Key type ._llcrypt unregistered [ 2077.431877] Key type ._llcrypt registered [ 2077.435487] Key type .llcrypt registered [ 2077.584716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.598347] alg: No test for adler32 (adler32-zlib) [ 2078.495793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2078.523488] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2084.789336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.149@tcp [ 2091.361947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.401095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2091.429759] LNet: Added LNI 192.168.202.49@tcp1 [8/256/0/180] [ 2092.041364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2092.523776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.149@tcp [ 2092.709972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2092.738598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2092.761851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2092.784643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2092.810978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2092.832622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2092.861464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2092.884615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2092.903777] LNet: 121285:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2092.908138] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.149@tcp [ 2092.919959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2092.944717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2092.970630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2092.991448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2093.016725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2093.042957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2093.581187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2093.787271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2093.842699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2093.890864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2093.938736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2093.960957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2093.992796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2094.021534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2094.078301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2094.091778] LNet: 121403:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2094.110685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2094.159914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2094.187752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2094.207977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2094.226834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2094.247373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2094.797820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.149@tcp [ 2095.028456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2095.059292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2095.078210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2095.098684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2095.131714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2095.172861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2095.204433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2095.241191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2095.264512] LNet: 121520:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2095.287757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2095.319698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2095.360448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2095.408370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2095.436915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2095.466283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2095.922117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2096.193911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2096.236152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2096.281485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2096.310803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2096.335902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2096.363148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2096.390673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2096.413617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2096.439615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2096.462827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2096.486762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2096.512739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2096.542642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2096.566817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2100.375463] LNet: 121718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.382817] LNetError: 121718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.390727] LNet: Removed LNI 192.168.202.49@tcp [ 2100.405091] LNet: Removed LNI 192.168.202.49@tcp1 [ 2100.713790] Key type .llcrypt unregistered [ 2100.717466] Key type ._llcrypt unregistered [ 2113.273569] Key type ._llcrypt registered [ 2113.277609] Key type .llcrypt registered [ 2114.725436] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:16:45 (1782487005) [ 2120.284268] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:16:51 (1782487011) [ 2126.120679] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:16:57 (1782487017) [ 2132.058417] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:17:03 (1782487023) [ 2137.939633] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:17:09 (1782487029) [ 2143.509697] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:17:14 (1782487034) [ 2148.814709] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:17:20 (1782487040) [ 2153.861123] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:17:25 (1782487045) [ 2159.555463] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:17:30 (1782487050) [ 2165.920602] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:17:37 (1782487057) [ 2171.924326] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:17:43 (1782487063) [ 2177.932236] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:17:49 (1782487069) [ 2183.645403] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:17:55 (1782487075) [ 2188.720020] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:18:00 (1782487080) [ 2193.716192] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:18:05 (1782487085) [ 2198.936744] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:18:10 (1782487090) [ 2204.225367] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:18:15 (1782487095) [ 2210.093364] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:18:21 (1782487101) [ 2210.424170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2210.438348] alg: No test for adler32 (adler32-zlib) [ 2211.379530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.486172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2211.507654] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2212.414195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.426453] LNet: 132424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.429932] LNetError: 132424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.448229] LNet: Removed LNI 192.168.202.49@tcp [ 2212.460365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.497633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2212.511223] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2217.386442] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:18:28 (1782487108) [ 2221.927578] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:18:33 (1782487113) [ 2227.084216] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:18:38 (1782487118) [ 2232.240992] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:18:43 (1782487123) [ 2236.588301] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:18:48 (1782487128) [ 2240.880560] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:18:52 (1782487132) [ 2245.087568] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:18:56 (1782487136) [ 2249.521875] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:19:01 (1782487141) [ 2254.110191] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:19:05 (1782487145) [ 2259.391713] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:19:10 (1782487150) [ 2263.863220] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:19:15 (1782487155) [ 2268.501835] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:19:19 (1782487159) [ 2273.665264] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2016 sec ========== 11:19:25 (1782487165) [ 2273.900591] LNet: 139885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.909537] LNetError: 139885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.937759] LNet: Removed LNI 192.168.202.49@tcp [ 2274.512511] Key type .llcrypt unregistered [ 2274.516320] Key type ._llcrypt unregistered [ 2282.329426] Key type ._llcrypt registered [ 2282.334302] Key type .llcrypt registered [ 2282.728373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.736651] alg: No test for adler32 (adler32-zlib) [ 2283.791676] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2284.016728] LNet: Added LNI 192.168.202.49@tcp [8/256/0/180] [ 2285.679432] Key type lgssc registered [ 2286.599638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.097142] Lustre: Mounted lustre-client [ 2329.712408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2333.379118] Lustre: Unmounted lustre-client [ 2345.807222] Key type lgssc unregistered [ 2345.966717] LNet: 142807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.970920] LNetError: 142807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.981617] LNet: Removed LNI 192.168.202.49@tcp [ 2347.443244] Key type .llcrypt unregistered [ 2347.445750] Key type ._llcrypt unregistered