[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 541056887 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003390] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007935] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010159] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.014078] *** VALIDATE selinux *** [ 0.021777] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025559] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028137] *** VALIDATE tmpfs *** [ 0.029000] *** VALIDATE proc *** [ 0.030259] *** VALIDATE cgroup *** [ 0.031012] *** VALIDATE cgroup2 *** [ 0.032236] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035028] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039679] debug: unmapping init [mem 0xffffffffb5859000-0xffffffffb5860fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042560] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043035] ... version: 2 [ 0.044018] ... bit width: 48 [ 0.045017] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047018] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.051345] rcu: Hierarchical SRCU implementation. [ 0.053641] smp: Bringing up secondary CPUs ... [ 0.054517] x86: Booting SMP configuration: [ 0.055020] .... node #0, CPUs: #1 #2 #3 [ 0.058464] smp: Brought up 1 node, 4 CPUs [ 0.060017] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.182240] node 0 deferred pages initialised in 119ms [ 0.185451] devtmpfs: initialized [ 0.189265] x86/mm: Memory block size: 128MB [ 0.191252] gcov: version magic: 0x41383552 [ 0.192751] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195056] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.196336] pinctrl core: initialized pinctrl subsystem [ 0.197356] [ 0.197668] ************************************************************* [ 0.198014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199075] ** ** [ 0.200013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201009] ** ** [ 0.202011] ** This means that this kernel is built to expose internal ** [ 0.203017] ** IOMMU data structures, which may compromise security on ** [ 0.204011] ** your system. ** [ 0.205008] ** ** [ 0.206009] ** If you see this message and you are not debugging the ** [ 0.207010] ** kernel, report this immediately to your vendor! ** [ 0.208011] ** ** [ 0.209007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210009] ************************************************************* [ 0.211703] NET: Registered protocol family 16 [ 0.212431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215467] cpuidle: using governor menu [ 0.217646] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219559] PCI: Using configuration type 1 for base access [ 0.221156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228225] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230177] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235116] cryptd: max_cpu_qlen set to 1000 [ 0.238252] ACPI: Added _OSI(Module Device) [ 0.240019] ACPI: Added _OSI(Processor Device) [ 0.241018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248867] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256191] ACPI: Interpreter enabled [ 0.257076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258015] ACPI: Using IOAPIC for interrupt routing [ 0.259108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270260] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.275185] acpiphp: Slot [2] registered [ 0.276133] acpiphp: Slot [5] registered [ 0.277126] acpiphp: Slot [6] registered [ 0.278115] acpiphp: Slot [3] registered [ 0.279098] acpiphp: Slot [4] registered [ 0.280131] acpiphp: Slot [7] registered [ 0.281090] acpiphp: Slot [8] registered [ 0.282079] acpiphp: Slot [9] registered [ 0.283082] acpiphp: Slot [10] registered [ 0.284108] acpiphp: Slot [11] registered [ 0.285072] acpiphp: Slot [12] registered [ 0.286097] acpiphp: Slot [13] registered [ 0.287088] acpiphp: Slot [14] registered [ 0.288111] acpiphp: Slot [15] registered [ 0.289072] acpiphp: Slot [16] registered [ 0.290088] acpiphp: Slot [17] registered [ 0.291060] acpiphp: Slot [18] registered [ 0.292065] acpiphp: Slot [19] registered [ 0.293300] acpiphp: Slot [20] registered [ 0.295148] acpiphp: Slot [21] registered [ 0.297127] acpiphp: Slot [22] registered [ 0.298180] acpiphp: Slot [23] registered [ 0.300118] acpiphp: Slot [24] registered [ 0.302204] acpiphp: Slot [25] registered [ 0.304263] acpiphp: Slot [26] registered [ 0.306155] acpiphp: Slot [27] registered [ 0.308262] acpiphp: Slot [28] registered [ 0.310267] acpiphp: Slot [29] registered [ 0.312288] acpiphp: Slot [30] registered [ 0.314145] acpiphp: Slot [31] registered [ 0.316138] PCI host bridge to bus 0000:00 [ 0.319040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332050] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337793] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340053] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345822] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.349478] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359013] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361071] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365630] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369906] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.377014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381073] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386669] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413237] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.443986] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.445437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.447215] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.448259] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.450190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.454678] iommu: Default domain type: Passthrough [ 0.455482] SCSI subsystem initialized [ 0.457117] ACPI: bus type USB registered [ 0.458093] usbcore: registered new interface driver usbfs [ 0.460078] usbcore: registered new interface driver hub [ 0.461071] usbcore: registered new device driver usb [ 0.463117] pps_core: LinuxPPS API ver. 1 registered [ 0.464008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.466093] PTP clock support registered [ 0.468139] EDAC MC: Ver: 3.0.0 [ 0.470136] PCI: Using ACPI for IRQ routing [ 0.472048] NetLabel: Initializing [ 0.474018] NetLabel: domain hash size = 128 [ 0.476012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.478087] NetLabel: unlabeled traffic allowed by default [ 0.481197] vgaarb: loaded [ 0.482405] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.485016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.493715] clocksource: Switched to clocksource kvm-clock [ 0.606627] VFS: Disk quotas dquot_6.6.0 [ 0.609248] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.613140] *** VALIDATE ramfs *** [ 0.614863] *** VALIDATE hugetlbfs *** [ 0.616633] pnp: PnP ACPI init [ 0.619242] pnp: PnP ACPI: found 6 devices [ 0.644912] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.648404] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.651058] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.653463] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.655960] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.658521] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.661687] NET: Registered protocol family 2 [ 0.663912] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.669164] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.673180] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.678038] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.681826] TCP: Hash tables configured (established 65536 bind 65536) [ 0.684502] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.687623] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690448] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.693063] NET: Registered protocol family 1 [ 0.696311] RPC: Registered named UNIX socket transport module. [ 0.698642] RPC: Registered udp transport module. [ 0.700496] RPC: Registered tcp transport module. [ 0.701651] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.704288] NET: Registered protocol family 44 [ 0.706554] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.709491] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.711743] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.715377] PCI: CLS 0 bytes, default 64 [ 0.717720] Unpacking initramfs... [ 2.126809] debug: unmapping init [mem 0xffff95c8fcc64000-0xffff95c8fffcffff] [ 2.130660] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.132685] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.135271] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.627849] Initialise system trusted keyrings [ 2.629470] Key type blacklist registered [ 2.631069] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.639443] zbud: loaded [ 2.642195] *** VALIDATE nfs *** [ 2.643300] *** VALIDATE nfs4 *** [ 2.644634] pstore: using deflate compression [ 2.647716] Platform Keyring initialized [ 2.738715] NET: Registered protocol family 38 [ 2.739965] Key type asymmetric registered [ 2.740903] Asymmetric key parser 'x509' registered [ 2.742086] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.746232] io scheduler mq-deadline registered [ 2.748307] io scheduler kyber registered [ 2.750428] io scheduler bfq registered [ 2.752700] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.755380] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.757888] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.759858] ACPI: Power Button [PWRF] [ 2.763443] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.767616] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.776162] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.803903] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.831064] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.836155] Non-volatile memory driver v1.3 [ 2.837613] Linux agpgart interface v0.103 [ 2.866590] virtio_blk virtio1: [vda] 145856 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.869310] vda: detected capacity change from 0 to 74678272 [ 2.885329] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.888228] vdb: detected capacity change from 0 to 1073741824 [ 2.894064] libphy: Fixed MDIO Bus: probed [ 2.913410] usbcore: registered new interface driver usbserial_generic [ 2.916594] usbserial: USB Serial support registered for generic [ 2.918868] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.924048] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.926436] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.929569] mousedev: PS/2 mouse device common for all mice [ 2.934601] rtc_cmos 00:05: RTC can wake from S4 [ 2.937672] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.939378] rtc_cmos 00:05: registered as rtc0 [ 2.947249] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.950303] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.951605] intel_pstate: CPU model not supported [ 2.957092] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.959047] hid: raw HID events driver (C) Jiri Kosina [ 2.962716] usbcore: registered new interface driver usbhid [ 2.964669] usbhid: USB HID core driver [ 2.966226] drop_monitor: Initializing network drop monitor service [ 2.968727] Initializing XFRM netlink socket [ 2.970804] NET: Registered protocol family 10 [ 2.975427] Segment Routing with IPv6 [ 2.977599] NET: Registered protocol family 17 [ 2.979712] mpls_gso: MPLS GSO support [ 2.986345] RAS: Correctable Errors collector initialized. [ 2.989315] AVX version of gcm_enc/dec engaged. [ 2.990985] AES CTR mode by8 optimization enabled [ 3.082216] sched_clock: Marking stable (3082191767, 0)->(3954045406, -871853639) [ 3.085832] registered taskstats version 1 [ 3.088386] Loading compiled-in X.509 certificates [ 3.091279] zswap: loaded using pool lzo/zbud [ 3.119803] Key type big_key registered [ 3.134938] Key type encrypted registered [ 3.136725] ima: No TPM chip found, activating TPM-bypass! [ 3.139172] ima: Allocated hash algorithm: sha1 [ 3.141192] ima: No architecture policies found [ 3.142856] evm: Initialising EVM extended attributes: [ 3.144305] evm: security.selinux [ 3.144978] evm: security.ima [ 3.145717] evm: security.capability [ 3.146823] evm: HMAC attrs: 0x1 [ 3.149770] rtc_cmos 00:05: setting system clock to 2026-08-14 21:26:15 UTC (1786742775) [ 3.157448] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 3.161114] debug: unmapping init [mem 0xffffffffb5582000-0xffffffffb5858fff] [ 3.174144] Write protecting the kernel read-only data: 28672k [ 3.177673] debug: unmapping init [mem 0xffffffffb3c03000-0xffffffffb3dfffff] [ 3.179908] debug: unmapping init [mem 0xffffffffb4514000-0xffffffffb45fffff] [ 3.210625] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.216157] systemd[1]: Detected virtualization kvm. [ 3.217734] systemd[1]: Detected architecture x86-64. [ 3.219046] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.248336] systemd[1]: No hostname configured. [ 3.249473] systemd[1]: Set hostname to . [ 3.251278] random: systemd: uninitialized urandom read (16 bytes read) [ 3.253654] systemd[1]: Initializing machine ID from random generator. [ 3.378463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.380492] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.383561] random: systemd: uninitialized urandom read (16 bytes read) [ 3.388273] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.396845] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ 3.447679] hrtimer: interrupt took 12686342 ns [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.128646] device-mapper: uevent: version 1.0.3 [ 4.132089] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.015214] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.231467] scsi host0: ata_piix [ 5.243353] scsi host1: ata_piix [ 5.245127] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.247638] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.654741] random: crng init done [ 9.656774] random: 7 urandom warning(s) missed due to ratelimiting [ 9.684081] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.540184] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.041594] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.135974] SELinux: Disabled at runtime. [ 12.202569] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.212709] systemd[1]: Detected virtualization kvm. [ 12.214734] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.804508] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.806973] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.812502] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.815586] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.817929] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.823899] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.832189] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ 12.969196] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.478685] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.811934] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.830809] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.252355] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.313522] EDAC sbridge: Ver: 1.1.2 [ 15.496119] Key type dns_resolver registered [ 15.832684] NFS: Registering the id_resolver key type [ 15.834507] Key type id_resolver registered [ 15.835964] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg242-client login: [ 47.303174] libcfs: loading out-of-tree module taints kernel. [ 47.374928] Key type ._llcrypt registered [ 47.376549] Key type .llcrypt registered [ 47.613108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.623330] alg: No test for adler32 (adler32-zlib) [ 48.806762] Lustre: Lustre: Build Version: 2.17.55_1_g696fb08 [ 49.843508] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 51.863188] Key type lgssc registered [ 54.057246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 202.521067] Lustre: Mounted lustre-client [ 208.368960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.378540] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.320579] Lustre: lustre-OST0000-osc-ffff95c9461da800: disconnect after 23s idle [ 231.797907] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing yml_node [ 236.843806] Lustre: DEBUG MARKER: Client: 2.17.55.1 [ 239.464943] Lustre: DEBUG MARKER: MDS: 2.17.55.1 [ 242.065640] Lustre: DEBUG MARKER: OSS: 2.17.55.1 [ 244.365706] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 17:30:14 EDT 2026 [ 264.838792] Lustre: Unmounted lustre-client [ 294.985260] Key type lgssc unregistered [ 295.292671] LNet: 8390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.302600] LNetError: 8390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.323880] LNet: Removed LNI 192.168.202.42@tcp [ 296.421197] Key type .llcrypt unregistered [ 296.423597] Key type ._llcrypt unregistered [ 320.443811] Key type ._llcrypt registered [ 320.454917] Key type .llcrypt registered [ 320.768709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.803696] alg: No test for adler32 (adler32-zlib) [ 321.702892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 321.747606] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 321.763773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 326.811658] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 326.911952] LNet: 9276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.917885] LNetError: 9276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.942007] LNet: Removed LNI 192.168.202.42@tcp [ 327.840127] Key type .llcrypt unregistered [ 327.844366] Key type ._llcrypt unregistered [ 335.200630] Key type ._llcrypt registered [ 335.207695] Key type .llcrypt registered [ 337.235307] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:31:47 (1786743107) [ 337.860575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.871666] alg: No test for adler32 (adler32-zlib) [ 338.833707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.951579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.973271] LNet: 9695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.993836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.079469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.106796] LNet: 9702:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 349.269492] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:31:59 (1786743119) [ 349.350557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.380534] LNet: 10275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.420328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.557462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 356.343488] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:32:07 (1786743127) [ 356.609789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.637832] LNet: 10871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.651952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.730782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.368562] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:32:14 (1786743134) [ 363.436663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.470412] LNet: 11448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.491130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.617744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 371.157092] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:32:21 (1786743141) [ 371.514611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.550828] LNet: 12040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.572301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.653290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 371.966240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 381.024764] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:32:31 (1786743151) [ 381.527358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.556889] LNet: 12630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.583697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.670246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 382.589363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 383.297028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.706242] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:32:43 (1786743163) [ 392.936038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.968610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.035973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 400.143257] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:32:51 (1786743171) [ 400.575796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.599200] LNet: 13812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.610952] LNet: 13812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 400.655655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.764735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 400.803532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.355308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 401.380276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 402.015474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 402.126925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.761731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 402.791389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.185681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 403.221149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.806598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 403.864460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.304706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 404.363308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.598829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 404.625480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 404.957699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 405.014581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 405.576608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 405.625726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 406.069866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 406.109729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 406.593878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 406.633756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.268970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.302875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 414.664368] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:33:06 (1786743186) [ 414.782226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.822984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.011158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 415.058954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 421.700397] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:33:13 (1786743193) [ 421.947388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.994339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.071521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 422.099603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.072921] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:33:19 (1786743199) [ 429.204858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.290987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.541281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 429.635834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 440.091591] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:33:30 (1786743210) [ 440.180342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.200462] LNet: 16250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.206442] LNet: 16250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 440.239839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.350551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.387448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 448.782567] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:33:39 (1786743219) [ 448.857605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.932399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.083422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 449.144386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 454.896952] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:33:46 (1786743226) [ 455.088087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.121959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.169633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 455.206283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 461.254408] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:33:52 (1786743232) [ 461.327859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.396601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.484186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.889903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.553859] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:34:01 (1786743241) [ 470.751946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.877074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.216801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.433793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.468948] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:34:10 (1786743250) [ 481.011028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.102713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.378316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.868409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.977100] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:34:24 (1786743264) [ 494.449707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.501099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.230611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.814363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.423679] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:34:34 (1786743274) [ 504.719756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.743125] LNet: 20505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.745154] LNet: 20505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 504.758035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.817100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.864360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.222093] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:34:43 (1786743283) [ 513.347531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.403574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.520703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 513.592574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.304423] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:34:53 (1786743293) [ 523.752070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.840657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.998371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 534.456905] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:35:05 (1786743305) [ 534.575897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.645345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.794674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 542.027564] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:35:12 (1786743312) [ 542.300881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.348800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.447327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 542.970458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 549.498827] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:35:20 (1786743320) [ 550.023631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.093592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.216616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 550.246540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 558.043685] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:35:28 (1786743328) [ 558.356720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.395398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.470419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 558.543266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 566.467549] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:35:36 (1786743336) [ 566.868602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.942955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.031899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 567.070635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 574.585866] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:35:45 (1786743345) [ 574.792608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.822925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.885872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 574.920097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.190566] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:35:54 (1786743354) [ 583.594188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.641980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.705842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 583.782271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 583.810753] LNetError: 25823:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 583.886591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 583.932936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 590.026373] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:36:01 (1786743361) [ 590.196713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.329737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.442360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oCLqjrtapj [ 590.582391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oCLqjrtapj [ 590.687983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oCLqjrtapj [ 598.626836] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:36:09 (1786743369) [ 598.724302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.803328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.884384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 598.940737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 598.994185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 604.852719] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:36:16 (1786743376) [ 605.434777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.486298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.587172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.623953] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 606.832610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 607.855462] Lustre: Lustre: Build Version: 2.17.55_1_g696fb08 [ 609.680373] Key type lgssc registered [ 610.858913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.538519] LNetError: 27585:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.202.42@tcp99 has no available nets [ 618.728081] Key type lgssc unregistered [ 619.138500] LNetError: 28692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.164240] LNet: Removed LNI 192.168.202.42@tcp [ 619.965236] Key type .llcrypt unregistered [ 619.970785] Key type ._llcrypt unregistered [ 629.984590] Key type ._llcrypt registered [ 629.991987] Key type .llcrypt registered [ 632.392940] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:36:41 (1786743401) [ 633.023160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.036418] alg: No test for adler32 (adler32-zlib) [ 634.239487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.354582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 639.557978] LNet: Message forwarding will be enabled in 2s [ 642.655177] LNet: Message forwarding enabled [ 643.247769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 643.262293] LNet: Message forwarding will stop in 2s to 6s [ 647.864189] LNet: Message forwarding disabled [ 647.878062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 649.169129] LNet: Message forwarding will be enabled in 2s [ 653.024566] LNet: Message forwarding enabled [ 657.649549] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:37:08 (1786743428) [ 657.765197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.803592] LNet: 30094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.898056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.006209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 665.627440] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:37:16 (1786743436) [ 665.979452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.005839] LNet: 30684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.042252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.109891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 666.141208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 666.196963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 666.241512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 666.308433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 666.362444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 666.402540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 666.462220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 666.529466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 666.597007] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 666.665938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 666.772397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 666.910338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 666.985521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 667.028554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 667.104508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 667.214820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 667.299435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 667.379548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 667.421963] LNetError: 30727:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 675.219923] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:37:26 (1786743446) [ 675.589629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.643833] LNet: 31301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.664622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.714701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.683852] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:37:34 (1786743454) [ 685.789602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg242-server [ 688.708603] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:37:38 (1786743458) [ 688.967074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.026867] LNet: 32232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.065487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.137044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.159238] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 689.780378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[43-47]@tcp [ 689.817164] LNet: 32365:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.874881] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 689.907090] LNetError: 32236:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.43@tcp has gone from up to down [ 690.931333] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 690.956194] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 691.938823] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 691.965727] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 694.002590] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 694.008158] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 697.057892] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.47@tcp [ 698.084542] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 698.099680] LNetError: 32236:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 13 previous similar messages [ 699.448928] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:37:49 (1786743469) [ 701.923400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg242-server [ 704.002777] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:37:55 (1786743475) [ 704.492763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.311237] LNetError: 32235:0:(router.c:1265:lnet_check_routers()) Failed to discover router 192.168.202.43@tcp [ 742.367405] LNetError: 32240:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.202.45 [ 742.390948] LNetError: Unexpected error -11 connecting to 192.168.202.45@tcp at host 192.168.202.45:988 [ 742.431237] LNetError: 32236:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.45@tcp has gone from up to down [ 742.452245] LNetError: 32236:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 742.467831] LNet: 33302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.483360] LNetError: 33302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.512551] LNet: Removed LNI 192.168.202.42@tcp [ 742.574155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.675862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.734135] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 743.633138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 743.658403] LNet: 33425:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 743.672267] LNet: 33425:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 743.705270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 743.737719] LNetError: 33311:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 743.762709] LNetError: 33311:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 20 previous similar messages [ 743.774673] LNetError: 33311:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.43@tcp has gone from up to down [ 754.985792] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:38:45 (1786743525) [ 755.180428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.615555] LNetError: 33317:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.202.45 [ 795.619347] LNetError: Unexpected error -11 connecting to 192.168.202.45@tcp at host 192.168.202.45:988 [ 796.064343] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 796.191709] LNet: 34012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.213403] LNetError: 34012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.292255] LNet: Removed LNI 192.168.202.42@tcp [ 797.353232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.495372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 797.600035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.670421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.751828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 797.835928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.935967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.060178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 798.201795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.265907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.374721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 798.533511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.618486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.741418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 798.901859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.969822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.100531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 799.242700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.299506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.402216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 799.566530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.668249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.790634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 799.906031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.997186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.103138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 800.217521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.263139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.398140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 800.565187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.660473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.794507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 800.945790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.021889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.095442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 810.503692] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:39:40 (1786743580) [ 813.304980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg242-server [ 815.263887] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:39:45 (1786743585) [ 817.858449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg242-server [ 819.630095] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:39:50 (1786743590) [ 819.772759] LNet: 35420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.776610] LNet: 35420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 820.520876] Key type .llcrypt unregistered [ 820.529102] Key type ._llcrypt unregistered [ 829.852410] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 830.125829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 830.129713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 830.135299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 832.801448] Key type ._llcrypt registered [ 832.804331] Key type .llcrypt registered [ 833.029894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.053374] alg: No test for adler32 (adler32-zlib) [ 834.104136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.211064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.247278] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 835.032180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 835.050618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 837.330894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 837.348914] LNet: Removed LNI 10.1.2.3@tcp [ 837.467419] LNet: 36147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.477756] LNetError: 36147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.495453] LNet: Removed LNI 192.168.202.42@tcp [ 838.230257] Key type .llcrypt unregistered [ 838.232506] Key type ._llcrypt unregistered [ 850.278965] Key type ._llcrypt registered [ 850.281822] Key type .llcrypt registered [ 851.723920] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:40:22 (1786743622) [ 852.328690] Key type .llcrypt unregistered [ 852.331155] Key type ._llcrypt unregistered [ 858.891835] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.946595] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.951057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.958591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.855432] Key type ._llcrypt registered [ 861.858238] Key type .llcrypt registered [ 862.126796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.149918] alg: No test for adler32 (adler32-zlib) [ 863.161761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.273402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.317885] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 864.240359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 864.266401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.956250] LNet: 37637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.960960] LNetError: 37637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.965882] LNet: Removed LNI 10.1.2.3@tcp [ 867.977600] LNet: Removed LNI 192.168.202.42@tcp [ 868.832575] Key type .llcrypt unregistered [ 868.834863] Key type ._llcrypt unregistered [ 880.812553] Key type ._llcrypt registered [ 880.821600] Key type .llcrypt registered [ 882.076416] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:40:52 (1786743652) [ 883.169444] Key type .llcrypt unregistered [ 883.171273] Key type ._llcrypt unregistered [ 890.614091] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.776883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.787462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.790969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.510611] Key type ._llcrypt registered [ 893.512651] Key type .llcrypt registered [ 893.671281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.679506] alg: No test for adler32 (adler32-zlib) [ 894.648811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.760550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 894.794592] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 897.531223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 897.583024] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.371402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 900.384657] LNet: Removed LNI 10.1.2.3@tcp [ 900.392976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 900.416941] LNet: Removed LNI 10.1.2.31@tcp [ 900.426877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 900.441791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.299479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 901.343806] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 902.744728] LNet: 39558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.756518] LNetError: 39558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 902.764061] LNet: Removed LNI 10.1.2.31@tcp [ 902.781772] LNet: Removed LNI 10.1.2.3@tcp [ 903.869656] Key type .llcrypt unregistered [ 903.871859] Key type ._llcrypt unregistered [ 916.147334] Key type ._llcrypt registered [ 916.149751] Key type .llcrypt registered [ 917.676140] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:41:28 (1786743688) [ 918.019908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.035863] alg: No test for adler32 (adler32-zlib) [ 919.032841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.115636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.139092] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 920.665402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.681804] LNet: 40512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.685672] LNetError: 40512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.695928] LNet: Removed LNI 192.168.202.42@tcp [ 920.710637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.762701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.42@tcp [ 920.783757] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 921.856103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.42@tcp [ 921.906499] LNet: Removed LNI 192.168.202.42@tcp [ 921.961816] LNet: 40647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.523541] Key type .llcrypt unregistered [ 922.525102] Key type ._llcrypt unregistered [ 936.324364] Key type ._llcrypt registered [ 936.325730] Key type .llcrypt registered [ 938.804919] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:41:48 (1786743708) [ 941.032808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg242-server [ 943.545665] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:41:53 (1786743713) [ 944.448368] Key type .llcrypt unregistered [ 944.451410] Key type ._llcrypt unregistered [ 951.286833] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 951.613833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 951.619387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 951.622978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 954.910532] Key type ._llcrypt registered [ 954.912616] Key type .llcrypt registered [ 955.388947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.444649] alg: No test for adler32 (adler32-zlib) [ 956.399071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.453833] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 957.287606] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 959.640552] LNet: 42483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.645892] LNetError: 42483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.651349] LNet: Removed LNI 192.168.202.42@tcp1 [ 959.666150] LNet: Removed LNI 10.1.2.3@tcp2 [ 960.363483] Key type .llcrypt unregistered [ 960.367621] Key type ._llcrypt unregistered [ 973.281570] Key type ._llcrypt registered [ 973.284541] Key type .llcrypt registered [ 974.526716] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:42:25 (1786743745) [ 975.282234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.315478] alg: No test for adler32 (adler32-zlib) [ 976.198400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.232662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 976.253046] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 977.137067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.171211] LNet: 43453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.177880] LNetError: 43453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.207089] LNet: Removed LNI 192.168.202.42@tcp [ 977.246566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.358548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 977.394868] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 984.252616] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:42:35 (1786743755) [ 984.389910] LNet: 44042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.395661] LNetError: 44042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.433356] LNet: Removed LNI 192.168.202.42@tcp [ 985.080912] Key type .llcrypt unregistered [ 985.085676] Key type ._llcrypt unregistered [ 991.467516] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.740073] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.759282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.770135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.386309] Key type ._llcrypt registered [ 994.387885] Key type .llcrypt registered [ 994.557482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.572096] alg: No test for adler32 (adler32-zlib) [ 995.636491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.749810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 995.820992] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 997.056336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 997.078414] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.308600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.375263] LNet: 44782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.396275] LNetError: 44782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.422056] LNet: Removed LNI 10.1.2.3@tcp [ 1000.459859] LNet: Removed LNI 192.168.202.42@tcp [ 1000.491488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.601433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1000.634212] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1000.643740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.140334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1001.186555] LNet: 44809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.205889] LNetError: 44809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1001.219631] LNetError: 44809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1001.239456] LNet: Removed LNI 10.1.2.3@tcp [ 1001.269673] LNet: Removed LNI 192.168.202.42@tcp [ 1001.353343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.469372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1001.524755] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1001.531871] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.638128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1002.676040] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1004.346700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.399628] LNet: 44972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.414680] LNetError: 44972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.427650] LNetError: 44972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1004.446823] LNet: Removed LNI 10.1.2.3@tcp [ 1004.452304] LNet: Removed LNI 192.168.202.42@tcp [ 1004.470857] LNet: Removed LNI 192.168.202.42@tcp2 [ 1004.526309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.671110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1004.738855] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1004.745756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.759153] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1016.363373] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:43:06 (1786743786) [ 1016.767531] LNet: 45571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.784234] LNetError: 45571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.794131] LNetError: 45571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1016.812431] LNet: Removed LNI 10.1.2.3@tcp [ 1016.819285] LNet: Removed LNI 192.168.202.42@tcp [ 1016.838248] LNet: Removed LNI 192.168.202.42@tcp2 [ 1017.838347] Key type .llcrypt unregistered [ 1017.840751] Key type ._llcrypt unregistered [ 1024.459171] Key type ._llcrypt registered [ 1024.460594] Key type .llcrypt registered [ 1024.760974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.791077] alg: No test for adler32 (adler32-zlib) [ 1025.744375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.830916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.861495] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1026.915779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.931948] LNet: 45965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.937687] LNetError: 45965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.958481] LNet: Removed LNI 192.168.202.42@tcp [ 1026.986946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.057700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1027.084186] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1027.820421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.854979] LNet: 45995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.869268] LNetError: 45995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.885229] LNet: Removed LNI 192.168.202.42@tcp [ 1027.932704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.044599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1028.105615] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1028.697572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.766315] LNet: Removed LNI 192.168.202.42@tcp [ 1028.810145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.954812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1029.004068] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1029.849754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.967732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1029.979459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1029.989461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.427142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.448913] LNet: 46105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.455507] LNet: 46105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1032.462295] LNetError: 46105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.468859] LNetError: 46105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1032.487335] LNet: Removed LNI 192.168.202.42@tcp [ 1032.505818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.586745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1032.619340] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1042.326838] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:43:32 (1786743812) [ 1042.551321] LNet: 46696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.574593] LNetError: 46696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.595427] LNet: Removed LNI 192.168.202.42@tcp [ 1043.713841] Key type .llcrypt unregistered [ 1043.716528] Key type ._llcrypt unregistered [ 1050.553735] Key type ._llcrypt registered [ 1050.555777] Key type .llcrypt registered [ 1050.929329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.953158] alg: No test for adler32 (adler32-zlib) [ 1051.986663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.017837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1052.031679] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1053.703746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.782707] LNet: 47101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.825751] LNetError: 47101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.892301] LNet: Removed LNI 192.168.202.42@tcp [ 1054.000902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.211778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1054.269442] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1064.290629] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:43:54 (1786743834) [ 1064.531742] LNet: 47686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.543405] LNetError: 47686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.602072] LNet: Removed LNI 192.168.202.42@tcp [ 1065.621542] Key type .llcrypt unregistered [ 1065.625601] Key type ._llcrypt unregistered [ 1074.817386] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1075.018958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1075.036615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1075.054997] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1078.106494] Key type ._llcrypt registered [ 1078.110534] Key type .llcrypt registered [ 1078.574722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.634519] alg: No test for adler32 (adler32-zlib) [ 1079.807726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.979251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.027478] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1081.471342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.485893] LNet: 48141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.493805] LNetError: 48141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.515315] LNet: Removed LNI 192.168.202.42@tcp [ 1081.554649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.644776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1081.675125] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1089.464294] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:44:20 (1786743860) [ 1089.867666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.894518] LNet: 48740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.899789] LNetError: 48740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.920747] LNet: Removed LNI 192.168.202.42@tcp [ 1090.000696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.189687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1099.765234] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:44:30 (1786743870) [ 1100.044534] LNet: 49319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.496895] Key type .llcrypt unregistered [ 1101.511989] Key type ._llcrypt unregistered [ 1109.617747] Key type ._llcrypt registered [ 1109.621139] Key type .llcrypt registered [ 1110.018481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.063466] alg: No test for adler32 (adler32-zlib) [ 1110.993804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.038529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1111.075564] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1112.181529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.199855] LNet: 49721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.207133] LNetError: 49721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.226669] LNet: Removed LNI 192.168.202.42@tcp [ 1112.250981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.309175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1112.326895] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1122.312435] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:44:52 (1786743892) [ 1122.969880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1123.017692] LNet: 50314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.030572] LNetError: 50314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.091645] LNet: Removed LNI 192.168.202.42@tcp [ 1123.141481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1123.248182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1132.563726] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:45:02 (1786743902) [ 1134.949396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1137.070741] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:45:07 (1786743907) [ 1138.922584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1140.855515] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:45:11 (1786743911) [ 1142.458377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1144.572758] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:45:15 (1786743915) [ 1146.245666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1148.055860] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:45:19 (1786743919) [ 1149.701901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1151.318771] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 17:45:22 (1786743922) [ 1151.391608] LNet: 52359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.099759] Key type .llcrypt unregistered [ 1152.102423] Key type ._llcrypt unregistered [ 1158.272902] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1158.417486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1158.424552] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1158.431457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1161.091784] Key type ._llcrypt registered [ 1161.093736] Key type .llcrypt registered [ 1161.377256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1161.403432] alg: No test for adler32 (adler32-zlib) [ 1162.334473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.420960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1162.459822] LNet: Added LNI 192.168.202.42@tcp [8/512/0/180] [ 1162.463590] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1162.465683] LNet: 52689:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1162.637478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1162.682713] LNet: 52708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.702986] LNetError: 52708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1162.714137] LNet: Removed LNI 10.1.2.3@tcp [ 1162.716727] LNetError: 52708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.739895] LNet: Removed LNI 192.168.202.42@tcp [ 1162.770556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.860288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1162.890325] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1163.002450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.030171] LNet: 52732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.042039] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.059930] LNet: Removed LNI 192.168.202.42@tcp [ 1163.085391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.195756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1163.245648] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1163.251683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1163.257102] LNet: 52740:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1163.380788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.395658] LNet: 52758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.400788] LNetError: 52758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.404624] LNet: Removed LNI 10.1.2.3@tcp [ 1163.406634] LNetError: 52758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.417693] LNet: Removed LNI 192.168.202.42@tcp [ 1163.429320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.482429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1163.506995] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1163.514093] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1163.517351] LNet: 52766:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1163.696511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.718259] LNet: 52792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.724671] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.730219] LNet: Removed LNI 10.1.2.3@tcp [ 1163.732630] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.749899] LNet: Removed LNI 192.168.202.42@tcp [ 1163.769457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.828689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1163.845743] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 1163.848840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1163.946935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.971524] LNet: 52820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.976111] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.992187] LNet: Removed LNI 10.1.2.3@tcp [ 1164.000399] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.038466] LNet: Removed LNI 192.168.202.42@tcp [ 1164.079660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.199690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1164.233636] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 1164.243494] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1164.250530] LNet: 52828:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1164.359885] LNet: Removed LNI 10.1.2.3@tcp [ 1164.385893] LNet: Removed LNI 192.168.202.42@tcp [ 1165.260943] Key type .llcrypt unregistered [ 1165.270314] Key type ._llcrypt unregistered [ 1177.667483] Key type ._llcrypt registered [ 1177.670774] Key type .llcrypt registered [ 1179.224159] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:45:49 (1786743949) [ 1179.603098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.618302] alg: No test for adler32 (adler32-zlib) [ 1180.535923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.607439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1180.636760] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1181.612508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.42 [ 1181.629863] LNet: 53805:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1181.638127] LNet: Removed LNI 192.168.202.42@tcp [ 1187.804784] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:45:58 (1786743958) [ 1188.071615] LNet: 54375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.076704] LNetError: 54375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.101104] LNet: Removed LNI 192.168.202.42@tcp [ 1188.705186] Key type .llcrypt unregistered [ 1188.711828] Key type ._llcrypt unregistered [ 1195.045390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1195.335330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1196.997656] Key type ._llcrypt registered [ 1197.000768] Key type .llcrypt registered [ 1197.180520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1197.197215] alg: No test for adler32 (adler32-zlib) [ 1198.651146] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1205.993137] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:46:17 (1786743977) [ 1206.367545] LNet: 55279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.383211] LNetError: 55279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1206.402601] LNet: Removed LNI 10.1.2.3@tcp [ 1207.225562] Key type .llcrypt unregistered [ 1207.227291] Key type ._llcrypt unregistered [ 1213.812105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1214.082122] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1214.098249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1214.102484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1216.857795] Key type ._llcrypt registered [ 1216.859885] Key type .llcrypt registered [ 1217.006960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.019987] alg: No test for adler32 (adler32-zlib) [ 1218.013526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1223.572482] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:46:34 (1786743994) [ 1225.430124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1226.882454] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:46:38 (1786743998) [ 1227.233283] LNet: 56464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.238745] LNetError: 56464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1227.258842] LNet: Removed LNI 10.1.2.3@tcp [ 1227.808580] Key type .llcrypt unregistered [ 1227.810508] Key type ._llcrypt unregistered [ 1235.540313] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1235.788364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1235.792472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1235.799868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1238.583617] Key type ._llcrypt registered [ 1238.587292] Key type .llcrypt registered [ 1238.776096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.788579] alg: No test for adler32 (adler32-zlib) [ 1239.758754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.841070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1250.785427] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:47:01 (1786744021) [ 1251.350211] LNet: 57694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.355414] LNetError: 57694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1251.372428] LNet: Removed LNI 10.1.2.3@tcp [ 1251.833621] Key type .llcrypt unregistered [ 1251.835338] Key type ._llcrypt unregistered [ 1257.762111] Key type ._llcrypt registered [ 1257.764665] Key type .llcrypt registered [ 1258.119362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.143571] alg: No test for adler32 (adler32-zlib) [ 1259.115541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.152654] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1266.986684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1267.833466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1267.872280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1267.903545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1267.916357] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1268.059580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.072280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.091904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.111656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.131886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.158136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.180225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.200402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.214350] LNet: 58147:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.231587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.254307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.281232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.317409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.350493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.380110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.383341] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1268.418928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1268.985678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1269.019625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1269.058409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1269.092294] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1269.097798] LNet: Skipped 1 previous similar message [ 1269.294678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.327923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.373713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.408681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.443377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.476509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.500614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.543269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.567452] LNet: 58256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.600976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.630439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.665315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.689360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.713905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.742439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.772526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.190615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1270.231942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1270.262880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1270.277721] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1270.466802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.509602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.536202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.570233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.618162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.646906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.699409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.727741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.740604] LNet: 58366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.753402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.772858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.791988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.811498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.835446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.868148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.899479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1271.368517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1271.405507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1271.437396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1271.677927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.705835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.728366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.754266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.777983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.807518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.836326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.873322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.911256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.947335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.979533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.007843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.043280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.108160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.161174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1272.704571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1272.750794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1272.788822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1272.808825] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1272.814342] LNet: Skipped 2 previous similar messages [ 1273.038120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.070691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1273.093670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.117991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.147697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.174173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.200874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.234987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.254033] LNet: 58584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1273.260925] LNet: 58584:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1273.276995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.293913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.316457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.337677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.365770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.400496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.433817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1273.992842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1274.034860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1274.072604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1274.309817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.365447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.415446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.492560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.520435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.573554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.620352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.662885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.701696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.738604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.761401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.825682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.861517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.910633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.989246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.346196] LNet: 58734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.355302] LNetError: 58734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.384936] LNet: Removed LNI 192.168.202.42@tcp [ 1275.879600] Key type .llcrypt unregistered [ 1275.881502] Key type ._llcrypt unregistered [ 1292.150972] Key type ._llcrypt registered [ 1292.153443] Key type .llcrypt registered [ 1293.498764] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:47:44 (1786744064) [ 1294.202793] Key type .llcrypt unregistered [ 1294.207134] Key type ._llcrypt unregistered [ 1299.783985] Key type ._llcrypt registered [ 1299.789865] Key type .llcrypt registered [ 1299.934725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1299.947537] alg: No test for adler32 (adler32-zlib) [ 1300.955182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1301.000308] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1308.762793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1322.307139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.324710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1322.342704] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1323.951400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1324.013568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1324.081638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1324.159198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1324.210476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1324.295099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1324.357579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1324.399548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1324.430565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1324.464136] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1324.473707] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1324.717433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1324.755218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1324.797406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1324.850168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1324.878117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1324.901803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1324.930730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1324.951686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1324.964192] LNet: 60200:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1324.973473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1325.003330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1325.029391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.062337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.088285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.119916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1325.203745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1325.873887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1325.918495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1325.949092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1325.979943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1326.013677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1326.045827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1326.082651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1326.127680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1326.168924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1326.188751] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1326.192768] LNet: Skipped 2 previous similar messages [ 1326.395639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.420308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.451069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.492557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.518497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.573102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.613147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.639542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.656619] LNet: 60321:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1326.673495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1326.703636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1326.725581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.749667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1326.780523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1326.808547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.832318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1327.323570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1327.355785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1327.387896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1327.423627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1327.459506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1327.492575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1327.524623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1327.565782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1327.625929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1327.647038] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1327.654987] LNet: Skipped 2 previous similar messages [ 1327.978053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.034298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.070973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.125090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.228349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.320985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.441639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.462418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.474220] LNet: 60444:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.490147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.533368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.555690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.581985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.608716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.652297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1328.692313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1329.518847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1329.575825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1329.631947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1329.689937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1329.744248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1329.806765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1329.855890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1329.886778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1329.909461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1329.926693] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1329.930228] LNet: Skipped 2 previous similar messages [ 1330.151914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.183820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.237846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.289305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.343971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.386422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.422645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.461277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.480870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.503661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.529400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.556446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.583426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.608740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.632454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1331.273294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1331.295985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1331.389734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1331.451071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1331.533641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1331.589944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1331.661811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1331.743916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1331.858213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1332.388669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.451973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.500961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.557864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.625094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.704988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.762733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.811697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.845756] LNet: 60686:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.872926] LNet: 60686:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1332.928891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.032870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.116888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.199870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.261294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.337046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.394336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1334.304951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1334.363568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1334.423790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1334.491632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1334.562800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1334.653522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1334.717074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1334.804940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1334.878893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1334.904966] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1334.909681] LNet: Skipped 7 previous similar messages [ 1335.309960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.379342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.441947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.518257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.584900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.661110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.768328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.850288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.040461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.125957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.241960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.368576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.417861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.490830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.569521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1344.502486] LNet: 60884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.530756] LNetError: 60884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.553592] LNet: Removed LNI 192.168.202.42@tcp [ 1345.577760] LNet: Removed LNI 192.168.202.42@tcp1 [ 1346.140710] Key type .llcrypt unregistered [ 1346.144073] Key type ._llcrypt unregistered [ 1366.402296] Key type ._llcrypt registered [ 1366.404375] Key type .llcrypt registered [ 1367.914238] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:48:58 (1786744138) [ 1369.161513] Key type .llcrypt unregistered [ 1369.163633] Key type ._llcrypt unregistered [ 1376.149304] Key type ._llcrypt registered [ 1376.157142] Key type .llcrypt registered [ 1376.432349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1376.471166] alg: No test for adler32 (adler32-zlib) [ 1377.402579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1377.467912] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1387.004444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1387.905752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1387.940349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1387.977043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1388.001437] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1388.259892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1388.302319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1388.336382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1388.379972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1388.421515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1388.464519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1388.492420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1388.558933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1388.579434] LNet: 62135:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1388.602675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1388.633541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1388.667330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1388.702976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1388.745980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1388.774806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1388.812783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1389.505191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1389.544217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1389.596577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1389.614748] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1389.901340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1389.935364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1389.966857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1389.997072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1390.017769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1390.064420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1390.105424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1390.166122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1390.195901] LNet: 62244:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1390.221924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1390.256975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1390.289858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1390.325425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1390.362976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1390.390537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1390.422516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1390.949983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1390.979063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1391.012366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1391.030930] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1391.234865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1391.260028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1391.282194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1391.306810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1391.338275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1391.360529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1391.389901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1391.429443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1391.443452] LNet: 62353:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.455718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1391.483400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1391.504110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.526801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.548029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1391.581103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1391.611865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1391.894377] LNet: 62394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.902403] LNetError: 62394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.917095] LNet: Removed LNI 192.168.202.42@tcp [ 1392.407515] Key type .llcrypt unregistered [ 1392.409276] Key type ._llcrypt unregistered [ 1410.773619] Key type ._llcrypt registered [ 1410.778377] Key type .llcrypt registered [ 1412.247928] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:49:42 (1786744182) [ 1413.178717] Key type .llcrypt unregistered [ 1413.180965] Key type ._llcrypt unregistered [ 1419.395620] Key type ._llcrypt registered [ 1419.399285] Key type .llcrypt registered [ 1419.584377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.596447] alg: No test for adler32 (adler32-zlib) [ 1420.547887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1420.571124] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1428.906511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1438.008486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.079919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1438.093836] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1439.525496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1439.576593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1439.644938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1439.710475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1439.773660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1439.806398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1439.842711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1439.920202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1439.975315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1439.989533] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1440.004422] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1440.354753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1440.404518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1440.437718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1440.461732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1440.529573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1440.555920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1440.594757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.622758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.641993] LNet: 63862:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1440.661865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.707513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.754606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.792871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.846066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.880907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1440.927767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1440.972894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1441.463279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1441.494290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1441.526374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1441.553097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1441.593762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1441.626594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1441.664783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1441.699906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1441.737924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1441.762980] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1441.766582] LNet: Skipped 2 previous similar messages [ 1442.010657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1442.034896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1442.073555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1442.105651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1442.135972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1442.157980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1442.185229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1442.212872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1442.233070] LNet: 63985:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1442.256399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1442.286924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1442.311971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.336444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.363695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.387220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1442.410821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.434900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.905582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1442.942812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1442.970734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1443.012660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1443.045904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1443.085670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1443.122637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1443.158977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1443.190584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1443.207672] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1443.212297] LNet: Skipped 1 previous similar message [ 1443.423932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.445614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1443.470342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.492834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.515724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.537869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.564676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.591751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.606387] LNet: 64108:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.625911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.650458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.676492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.703782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1443.729349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.759517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1443.794564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1443.822418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1448.417624] LNet: 64186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.422164] LNetError: 64186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.428153] LNet: Removed LNI 192.168.202.42@tcp [ 1448.446765] LNet: Removed LNI 192.168.202.42@tcp1 [ 1448.787600] Key type .llcrypt unregistered [ 1448.789693] Key type ._llcrypt unregistered [ 1464.209400] Key type ._llcrypt registered [ 1464.214488] Key type .llcrypt registered [ 1465.614716] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:50:36 (1786744236) [ 1466.934457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1468.778738] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:50:39 (1786744239) [ 1469.548495] Key type .llcrypt unregistered [ 1469.550474] Key type ._llcrypt unregistered [ 1475.816546] Key type ._llcrypt registered [ 1475.818274] Key type .llcrypt registered [ 1476.069231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1476.090262] alg: No test for adler32 (adler32-zlib) [ 1476.961454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1476.999624] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1485.108295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1485.798129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1485.834808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1485.865483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1485.881657] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1486.102791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.125224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1486.186879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.225194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.240072] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1486.258211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.284553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.316908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.340233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.354580] LNet: 65729:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.369346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.391646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.419808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.448612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.486720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.536845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.752225] LNet: 65768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.759300] LNetError: 65768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.784787] LNet: Removed LNI 192.168.202.42@tcp [ 1488.273548] Key type .llcrypt unregistered [ 1488.276070] Key type ._llcrypt unregistered [ 1506.406967] Key type ._llcrypt registered [ 1506.409357] Key type .llcrypt registered [ 1506.599876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.623204] alg: No test for adler32 (adler32-zlib) [ 1507.580277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1507.619300] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1513.747618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1521.324724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.412594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1521.451945] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1523.176496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1523.204904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1523.223483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1523.244900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1523.290762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1523.324138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1523.356229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1523.416771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1523.461872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1523.499489] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1523.813034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1523.850960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1523.910857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1523.956582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1523.987943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1523.999206] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1523.999225] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1524.023325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1524.043649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1524.077752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1524.091695] LNet: 66647:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.114595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1524.159699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1524.205872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1524.263462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1524.308170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1524.365840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1528.240621] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1528.504948] LNet: 66721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.508991] LNetError: 66721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.514625] LNet: Removed LNI 192.168.202.42@tcp [ 1528.522821] LNet: Removed LNI 192.168.202.42@tcp1 [ 1528.896906] Key type .llcrypt unregistered [ 1528.914282] Key type ._llcrypt unregistered [ 1547.949727] Key type ._llcrypt registered [ 1547.951759] Key type .llcrypt registered [ 1549.476126] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:52:00 (1786744320) [ 1550.059448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.077416] alg: No test for adler32 (adler32-zlib) [ 1551.076385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.171528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.194851] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1551.973372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1552.003783] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1552.919404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1553.022069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1553.098764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1553.152724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1553.200464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1553.226691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1553.252358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1553.265844] LNetError: 67820:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1554.207781] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1554.208140] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1556.257751] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1556.905980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1560.356508] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1560.713454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1560.780625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1560.863193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.886258] LNet: 67927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.890062] LNetError: 67927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.893410] LNet: Removed LNI 192.168.202.42@tcp [ 1560.910985] LNet: Removed LNI 192.168.202.42@tcp1 [ 1560.956968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.041356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.079462] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1562.133328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1562.163607] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1563.258944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1563.317397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1563.410799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1563.502561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1563.553161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1563.584183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1563.640084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1563.683562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1563.718845] LNetError: 68200:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1564.065538] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1567.302417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1570.207515] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1570.212121] LNet: Skipped 1 previous similar message [ 1575.346425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1575.394556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1575.436930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.852419] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:52:32 (1786744352) [ 1582.145458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.170414] LNet: 68917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.181414] LNetError: 68917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.188272] LNetError: 68917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1582.194330] LNet: Removed LNI 192.168.202.42@tcp [ 1582.203447] LNet: Removed LNI 192.168.202.42@tcp1 [ 1582.237829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.329787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.362180] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1582.982380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1583.001318] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1583.699735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1583.778559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1583.815966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1583.856143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1583.899707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1583.960053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1584.008467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1584.106269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1584.351556] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.42@tcp [ 1585.375549] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1585.380356] LNet: Skipped 1 previous similar message [ 1600.682368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1600.900902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1600.924866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1600.955476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 500 [ 1601.078843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1607.869345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1607.946414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.996981] LNet: 69417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.006846] LNetError: 69417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.012272] LNetError: 69417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.055168] LNet: Removed LNI 192.168.202.42@tcp [ 1609.067790] LNet: Removed LNI 192.168.202.42@tcp1 [ 1609.133782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.230269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.266735] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1610.192949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.218783] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1611.056807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1611.109152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1611.198141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1611.242631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1611.302781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1611.353165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1611.416529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1611.463977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1612.255470] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1612.264354] LNet: Skipped 3 previous similar messages [ 1612.869913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1623.240592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1623.290419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1623.355730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.992934] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:53:21 (1786744401) [ 1631.554686] LNet: 70464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.560769] LNetError: 70464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.564046] LNetError: 70464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1631.570855] LNet: Removed LNI 192.168.202.42@tcp [ 1631.584724] LNet: Removed LNI 192.168.202.42@tcp1 [ 1632.148352] Key type .llcrypt unregistered [ 1632.155278] Key type ._llcrypt unregistered [ 1638.227763] Key type ._llcrypt registered [ 1638.235551] Key type .llcrypt registered [ 1638.658643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1638.697656] alg: No test for adler32 (adler32-zlib) [ 1639.612864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1639.636212] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1650.691679] Lustre: DEBUG MARKER: Initial discovery [ 1650.811531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1653.868313] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1654.068988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1654.173114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1654.254984] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1654.310826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1654.376786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -l 3 [ 1654.410886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -e local_error [ 1654.452787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m PUT -l 6 [ 1655.007268] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1656.589523] Lustre: DEBUG MARKER: Force oleg242-client.virtnet to discover 192.168.202.142@tcp (in background) [ 1656.894734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1658.906496] Lustre: DEBUG MARKER: Force oleg242-server to discover 192.168.202.42@tcp [ 1660.001612] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1661.439427] Lustre: DEBUG MARKER: Wait for 71230 [ 1662.975880] Lustre: DEBUG MARKER: Finished wait on 71230 [ 1663.047812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1663.084768] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1663.096267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1663.404479] LNet: 71678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.415665] LNetError: 71678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.430355] LNet: Removed LNI 192.168.202.42@tcp [ 1663.729730] Key type .llcrypt unregistered [ 1663.736696] Key type ._llcrypt unregistered [ 1680.233485] Key type ._llcrypt registered [ 1680.235299] Key type .llcrypt registered [ 1681.441784] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:54:12 (1786744452) [ 1682.160412] Key type .llcrypt unregistered [ 1682.163867] Key type ._llcrypt unregistered [ 1687.888821] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1687.988370] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1688.000065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1688.006295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1690.576265] Key type ._llcrypt registered [ 1690.578123] Key type .llcrypt registered [ 1690.787797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1690.804392] alg: No test for adler32 (adler32-zlib) [ 1691.693343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.754517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.780383] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1692.479447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1692.505036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1701.044625] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:54:32 (1786744472) [ 1701.356437] LNet: 73757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.361295] LNetError: 73757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1701.366348] LNet: Removed LNI 10.1.2.3@tcp [ 1701.375099] LNet: Removed LNI 192.168.202.42@tcp [ 1701.998519] Key type .llcrypt unregistered [ 1702.000897] Key type ._llcrypt unregistered [ 1707.636878] Key type ._llcrypt registered [ 1707.643432] Key type .llcrypt registered [ 1707.808344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1707.819428] alg: No test for adler32 (adler32-zlib) [ 1708.734297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.801282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.831361] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1709.518419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1709.536261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1710.452229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.42@tcp [ 1711.519827] LNetError: Unexpected error -22 connecting to 192.168.202.42@tcp at host 192.168.202.42:988 [ 1717.435212] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:54:48 (1786744488) [ 1717.704922] LNet: 74889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.712221] LNetError: 74889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1717.725522] LNet: Removed LNI 10.1.2.3@tcp [ 1717.752594] LNet: Removed LNI 192.168.202.42@tcp [ 1718.494439] Key type .llcrypt unregistered [ 1718.496702] Key type ._llcrypt unregistered [ 1726.059662] Key type ._llcrypt registered [ 1726.062840] Key type .llcrypt registered [ 1726.265558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1726.279649] alg: No test for adler32 (adler32-zlib) [ 1727.205744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.252519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1727.272236] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1728.056620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1728.074730] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1729.008891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.42@tcp1 --nid 192.168.202.42@tcp2 [ 1734.260458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp2 [ 1734.279132] LNetError: 75727:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.42@tcp1 to 192.168.202.42@tcp2 - Route table may be misconfigured [ 1734.297596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp2 192.168.202.42@tcp1 [ 1741.525912] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:55:12 (1786744512) [ 1741.675602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.768624] LNet: 76295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.794512] LNetError: 76295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.811548] LNet: Removed LNI 192.168.202.42@tcp1 [ 1742.892363] LNet: Removed LNI 192.168.202.42@tcp2 [ 1742.946700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.020078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.062926] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1743.978050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1744.019589] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1745.053637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1745.103785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.42@tcp [ 1745.151960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp1 [ 1745.218925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp -e network_timeout [ 1745.273766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -e network_timeout [ 1745.321029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -e network_timeout [ 1745.365360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -e network_timeout [ 1745.424049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1745.449859] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1745.480982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1745.651765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1745.847896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1746.079902] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1753.454971] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:55:24 (1786744524) [ 1753.852796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.870982] LNet: 77164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.884865] LNetError: 77164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.891668] LNetError: 77164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1753.902733] LNet: Removed LNI 192.168.202.42@tcp [ 1754.920847] LNet: Removed LNI 192.168.202.42@tcp1 [ 1754.975485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.036812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1755.329542] LNet: 77201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.735533] Key type .llcrypt unregistered [ 1755.741440] Key type ._llcrypt unregistered [ 1768.426995] Key type ._llcrypt registered [ 1768.429277] Key type .llcrypt registered [ 1769.886612] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:55:40 (1786744540) [ 1770.596408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.615288] alg: No test for adler32 (adler32-zlib) [ 1771.632613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.329555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1775.186637] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:55:46 (1786744546) [ 1775.456929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.480957] LNet: 78300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.508243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.559631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.580757] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1776.446392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1776.466875] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1777.249725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1777.311960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp1 [ 1777.361670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp1 [ 1785.028697] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:55:55 (1786744555) [ 1787.124143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg242-server [ 1789.324345] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:55:59 (1786744559) [ 1791.532114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg242-server [ 1793.375892] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:56:04 (1786744564) [ 1795.665951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg242-server [ 1797.085417] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:56:08 (1786744568) [ 1799.297429] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg242-server [ 1800.964704] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:56:11 (1786744571) [ 1802.936892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg242-server [ 1804.611478] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:56:15 (1786744575) [ 1806.445207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg242-server [ 1808.560548] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:56:19 (1786744579) [ 1810.548460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg242-server [ 1812.672767] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:56:23 (1786744583) [ 1814.996103] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg242-server [ 1816.804466] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:56:27 (1786744587) [ 1818.646983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg242-server [ 1820.316023] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:56:31 (1786744591) [ 1820.536286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.569558] LNet: 82176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.575308] LNetError: 82176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.579711] LNet: Removed LNI 192.168.202.42@tcp [ 1821.625683] LNet: Removed LNI 192.168.202.42@tcp1 [ 1821.659267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.705920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.734612] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1822.606625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1822.707194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1822.935634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.957390] LNet: 82324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.991049] LNetError: 82324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.004015] LNetError: 82324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1823.042616] LNet: Removed LNI 192.168.202.42@tcp [ 1823.070389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.164221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1823.197421] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1823.892148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1824.139194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.181895] LNetError: 82470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.208220] LNet: Removed LNI 192.168.202.42@tcp [ 1824.257491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.310865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.327033] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1825.047728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1825.131391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1826.479612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.500238] LNet: 82626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.504191] LNet: 82626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1826.511688] LNetError: 82626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.557809] LNet: Removed LNI 192.168.202.42@tcp [ 1827.584248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.681559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1827.754396] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1828.530575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1828.754968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.805719] LNet: Removed LNI 192.168.202.42@tcp [ 1829.867277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.951049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.971624] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1830.765313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1830.832587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1831.033779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.059445] LNet: 82919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.063972] LNet: 82919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1831.073448] LNetError: 82919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.075766] LNetError: 82919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1832.111341] LNet: Removed LNI 192.168.202.42@tcp [ 1832.171501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.244557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1832.270431] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1832.992846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1833.256807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.350665] LNet: Removed LNI 192.168.202.42@tcp [ 1834.375990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.488170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.561959] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1835.585300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1835.652545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1835.925914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.966938] LNet: Removed LNI 192.168.202.42@tcp [ 1836.987247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.079362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1837.103986] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1838.140097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1838.323959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.398960] LNet: Removed LNI 192.168.202.42@tcp [ 1839.458728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.523884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.555977] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1840.589588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1840.703989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1840.962832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.984907] LNet: 83504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.995385] LNet: 83504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1841.020419] LNetError: 83504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.030110] LNetError: 83504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1841.047159] LNet: Removed LNI 192.168.202.42@tcp [ 1841.072161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.158363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1841.184984] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1842.037697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1843.633076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.718582] LNet: Removed LNI 192.168.202.42@tcp [ 1844.772265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.823939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.854443] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1845.928311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1846.102140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1848.863047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.964935] LNet: Removed LNI 192.168.202.42@tcp [ 1850.011931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.083029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1850.118342] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1851.513493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1851.901460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.968210] LNet: Removed LNI 192.168.202.42@tcp [ 1853.019251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.148475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1853.190835] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1854.781856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1854.875947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.947330] LNet: Removed LNI 192.168.202.42@tcp [ 1854.970260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.053251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.098642] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1856.194905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1856.239608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.255955] LNet: Removed LNI 192.168.202.42@tcp [ 1856.266833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.301801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1856.359884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.416590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.507130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1856.665250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.698788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.737916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1856.833787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.897167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.075545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1857.197783] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1858.412362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.443271] LNet: 84410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.452912] LNet: 84410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1858.468573] LNetError: 84410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.478781] LNetError: 84410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1858.507669] LNet: Removed LNI 192.168.202.42@tcp [ 1858.534227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.596739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1858.669418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1858.702817] LNet: 84422:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1866.824411] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:57:17 (1786744637) [ 1866.981406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.050631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.163203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1867.190009] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1868.270785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.328522] LNet: Removed LNI 192.168.202.42@tcp231 [ 1868.352498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.461657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1868.537615] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1868.544544] LNet: 85144:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.211785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1869.247767] LNet: Removed LNI 192.168.202.42@tcp231 [ 1869.266847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1869.317086] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1870.612828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.674854] LNet: Removed LNI 192.168.202.42@tcp231 [ 1870.707455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.770378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1870.794769] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1870.800376] LNet: 85296:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1871.262395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1871.299558] LNet: Removed LNI 192.168.202.42@tcp231 [ 1871.328839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1871.363407] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1872.726666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.755524] LNet: Removed LNI 192.168.202.42@tcp231 [ 1872.796695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.893846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1872.931213] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1872.935527] LNet: 85448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1873.474289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1873.521042] LNet: Removed LNI 192.168.202.42@tcp231 [ 1873.534859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1873.553697] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1874.484683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.527589] LNet: Removed LNI 192.168.202.42@tcp231 [ 1874.570311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.676444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1874.694285] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1875.174655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1875.205169] LNet: Removed LNI 192.168.202.42@tcp231 [ 1875.219792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1875.249552] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1882.647980] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:57:33 (1786744653) [ 1883.152602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.201309] LNet: Removed LNI 192.168.202.42@tcp231 [ 1883.224695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.303242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1883.318978] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1883.959137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1883.996643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1884.019930] LNet: Removed LNI 192.168.202.42@tcp232 [ 1884.037386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1884.055928] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1884.924420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1884.973568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1885.010194] LNet: Removed LNI 192.168.202.42@tcp232 [ 1885.029921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1885.061745] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1885.836301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1885.864717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1885.892301] LNet: Removed LNI 192.168.202.42@tcp232 [ 1885.910546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1885.932917] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1886.742724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1886.798988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1886.843462] LNet: Removed LNI 192.168.202.42@tcp232 [ 1886.859077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1893.848955] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:57:44 (1786744664) [ 1894.305907] LNet: 87437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.309500] LNet: 87437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1894.843604] Key type .llcrypt unregistered [ 1894.845809] Key type ._llcrypt unregistered [ 1901.892477] Key type ._llcrypt registered [ 1901.894273] Key type .llcrypt registered [ 1902.152602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1902.162688] alg: No test for adler32 (adler32-zlib) [ 1903.156454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1903.189453] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1910.969839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1919.882530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.948145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1919.991122] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1921.372376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1921.414782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1921.453918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1921.496454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1921.530337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1921.574854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1921.617868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1921.660894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1921.695084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1921.722503] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1921.727787] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1921.972236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1921.994241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1922.020045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1922.045797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1922.072851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1922.117885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1922.150105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1922.183253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1922.197703] LNet: 88099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1922.214197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1922.242217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1922.273269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1922.299371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1922.336134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1922.364093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.822972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1922.853483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1922.878786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1922.902482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1922.930285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1922.960740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1922.992783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1923.027990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1923.086044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1923.128831] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1923.590917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1923.660382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1923.715541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1923.785453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1923.835554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1923.863544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1923.897251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1923.927287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1923.947119] LNet: 88219:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.985810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1924.034280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1924.096434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1924.132968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1924.212814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1924.269344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1925.134857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1925.171753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1925.211219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1925.235665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1925.274728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1925.310726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1925.350718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1925.378541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1925.408238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1925.434871] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1925.694878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1925.727817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1925.780796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1925.817936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1925.852954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1925.890703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1925.929936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1925.966875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1925.988855] LNet: 88339:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1926.014751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1926.047412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1926.082256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1926.120468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1926.165615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1926.203926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1926.822152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1926.868349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1926.907967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1926.944347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1927.004231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1927.056501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1927.095793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1927.153847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1927.191600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1927.555439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1927.581762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1927.633675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1927.668731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1927.726276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1927.770964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1927.829791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1927.893285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1927.930272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1927.994290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1928.047065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1928.093317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1928.140047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1928.175858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1928.890878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1928.946943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1928.997210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1929.070877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1929.123220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1929.158664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1929.200142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1929.265864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1929.319799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1929.340421] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1929.346122] LNet: Skipped 1 previous similar message [ 1929.576622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1929.609935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1929.644775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1929.669713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1929.694736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1929.724891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1929.753292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1929.781799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1929.797934] LNet: 88580:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1929.802811] LNet: 88580:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1929.819767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1929.846689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1929.875934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1929.905849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1929.936978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1929.995570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1930.778697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1930.839938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1930.900026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1930.941583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1930.993302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1931.045787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1931.093393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1931.134654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1931.195773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1931.221833] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1931.462346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1931.485983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1931.510326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1931.554847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1931.585804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1931.617869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1931.649048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1931.679401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1931.721663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1931.765353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1931.800725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1931.837537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1931.904613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1931.938643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1937.957574] LNet: 88774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.972750] LNetError: 88774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.979026] LNet: Removed LNI 192.168.202.42@tcp [ 1937.993080] LNet: Removed LNI 192.168.202.42@tcp1 [ 1938.707856] Key type .llcrypt unregistered [ 1938.710135] Key type ._llcrypt unregistered [ 1961.743686] Key type ._llcrypt registered [ 1961.745307] Key type .llcrypt registered [ 1963.261352] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:58:53 (1786744733) [ 1964.451403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.513837] alg: No test for adler32 (adler32-zlib) [ 1965.554180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.619556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.642704] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1966.735785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1966.779279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1966.810643] LNetError: 89752:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1966.911209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.42@tcp [ 1967.057159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1975.622637] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:59:06 (1786744746) [ 1975.793217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.871696] LNet: 90336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.895580] LNetError: 90336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.919534] LNet: Removed LNI 192.168.202.42@tcp [ 1975.954440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.059284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.083310] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1977.044217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1977.082651] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1978.648850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.42@tcp [ 1978.694148] LNetError: 90609:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1979.104783] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1979.108518] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1980.181769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.42@tcp [ 1989.674584] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:59:20 (1786744760) [ 1991.943170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg242-server [ 1993.792599] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:59:24 (1786744764) [ 1995.634255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg242-server [ 1997.524919] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:59:28 (1786744768) [ 1999.692654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg242-server [ 2001.277209] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:59:32 (1786744772) [ 2001.491680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.520743] LNet: 92233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.533413] LNetError: 92233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.547312] LNet: Removed LNI 192.168.202.42@tcp [ 2001.562083] LNet: Removed LNI 192.168.202.42@tcp1 [ 2001.590493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.654331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.690961] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2002.484980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2002.575876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2002.623776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2002.711067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2002.818612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2002.870288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2010.334941] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:59:41 (1786744781) [ 2010.471953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.505103] LNet: 92952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.543315] LNetError: 92952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.554382] LNetError: 92952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2010.599660] LNet: Removed LNI 192.168.202.42@tcp [ 2010.644779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.750608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2010.779101] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2018.611031] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:59:49 (1786744789) [ 2018.975292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.994653] LNet: 93661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.002555] LNetError: 93661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.016215] LNet: Removed LNI 192.168.202.42@tcp [ 2019.048385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.109571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.157540] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2026.360815] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:59:57 (1786744797) [ 2028.181319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2030.290515] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:00:01 (1786744801) [ 2030.904231] LNet: 94697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.910685] LNetError: 94697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.928736] LNet: Removed LNI 192.168.202.42@tcp [ 2031.448195] Key type .llcrypt unregistered [ 2031.450053] Key type ._llcrypt unregistered [ 2037.139390] Key type ._llcrypt registered [ 2037.146642] Key type .llcrypt registered [ 2037.429779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2037.463961] alg: No test for adler32 (adler32-zlib) [ 2038.401902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2038.421445] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2045.955916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2050.659279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.142@tcp [ 2051.198847] LNet: 95114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.206671] LNetError: 95114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.241299] LNet: Removed LNI 192.168.202.42@tcp [ 2052.038616] Key type .llcrypt unregistered [ 2052.040914] Key type ._llcrypt unregistered [ 2067.360695] Key type ._llcrypt registered [ 2067.362305] Key type .llcrypt registered [ 2068.442871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2070.144148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2071.859492] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:00:42 (1786744842) [ 2073.611096] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2075.921482] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:00:46 (1786744846) [ 2078.155851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg242-server [ 2080.133600] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:00:51 (1786744851) [ 2082.515636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg242-server [ 2084.427280] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:00:55 (1786744855) [ 2086.066129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2086.085698] alg: No test for adler32 (adler32-zlib) [ 2087.219508] Lustre: Lustre: Build Version: 2.17.55_1_g696fb08 [ 2087.527708] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2089.256237] Key type lgssc registered [ 2090.679989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.870700] Key type lgssc unregistered [ 2116.240900] LNet: 98303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.247651] LNetError: 98303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.267389] LNet: Removed LNI 192.168.202.42@tcp [ 2117.050652] Key type .llcrypt unregistered [ 2117.054808] Key type ._llcrypt unregistered [ 2132.582139] Key type ._llcrypt registered [ 2132.584627] Key type .llcrypt registered [ 2134.107789] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:01:44 (1786744904) [ 2134.447124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.465881] alg: No test for adler32 (adler32-zlib) [ 2135.469413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.580514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2135.631142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2135.677491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2135.749743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2143.066916] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:01:53 (1786744913) [ 2143.225666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.238554] LNet: 99719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.254432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.303532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.327836] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2144.085365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.106466] LNet: 99863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.111171] LNetError: 99863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.120682] LNet: Removed LNI 192.168.202.42@tcp [ 2144.135624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.177655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2144.214519] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2144.875655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.905968] LNetError: 99891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.923601] LNet: Removed LNI 192.168.202.42@tcp [ 2144.955536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.029213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2145.061129] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2152.246925] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:02:03 (1786744923) [ 2152.431632] LNet: 100481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.434617] LNet: 100481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2152.438624] LNetError: 100481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.449577] LNet: Removed LNI 192.168.202.42@tcp [ 2153.106624] Key type .llcrypt unregistered [ 2153.109643] Key type ._llcrypt unregistered [ 2159.090417] Key type ._llcrypt registered [ 2159.097560] Key type .llcrypt registered [ 2159.312353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2159.333853] alg: No test for adler32 (adler32-zlib) [ 2160.194840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.259418] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2160.263681] LNet: *** cfs_fail_loc=e003, val=100*** [ 2160.944833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2160.969249] LNet: *** cfs_fail_loc=e003, val=100*** [ 2160.972253] LNet: Skipped 2 previous similar messages [ 2160.987203] Ping reply received with mapping: [ 2160.994950] found metadata reply NI[192.168.202.42@tcp] [ 2160.997966] 2a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2161.007550] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2161.010455] 5a 5a 5a 5a 5a 5a 5a 5a [ 2161.241316] LNet: 100878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.244520] LNetError: 100878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.261973] LNet: Removed LNI 192.168.202.42@tcp [ 2161.986412] Key type .llcrypt unregistered [ 2161.988773] Key type ._llcrypt unregistered [ 2174.313577] Key type ._llcrypt registered [ 2174.315817] Key type .llcrypt registered [ 2175.738650] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:02:26 (1786744946) [ 2176.194547] Key type .llcrypt unregistered [ 2176.195830] Key type ._llcrypt unregistered [ 2183.123941] Key type ._llcrypt registered [ 2183.127250] Key type .llcrypt registered [ 2183.347093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.365524] alg: No test for adler32 (adler32-zlib) [ 2184.516754] LNetError: 101924:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2184.908145] Key type .llcrypt unregistered [ 2184.912735] Key type ._llcrypt unregistered [ 2190.411703] Key type ._llcrypt registered [ 2190.413897] Key type .llcrypt registered [ 2191.945402] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:02:42 (1786744962) [ 2192.737507] Key type .llcrypt unregistered [ 2192.739585] Key type ._llcrypt unregistered [ 2198.945132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2198.955857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2201.901237] Key type ._llcrypt registered [ 2201.903344] Key type .llcrypt registered [ 2202.296245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2202.336332] alg: No test for adler32 (adler32-zlib) [ 2203.363652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.490165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2216.936283] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:03:07 (1786744987) [ 2217.248557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.314711] LNet: 103704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.322890] LNetError: 103704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2217.346261] LNet: Removed LNI 10.1.2.3@tcp [ 2217.364447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.959958] Key type .llcrypt unregistered [ 2217.962054] Key type ._llcrypt unregistered [ 2225.080215] Key type ._llcrypt registered [ 2225.082741] Key type .llcrypt registered [ 2225.324904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2225.337240] alg: No test for adler32 (adler32-zlib) [ 2226.412696] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2226.556253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2226.966819] LNet: 103993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.979187] LNetError: 103993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.015382] LNet: Removed LNI 192.168.202.42@tcp [ 2227.834800] Key type .llcrypt unregistered [ 2227.841453] Key type ._llcrypt unregistered [ 2241.700121] Key type ._llcrypt registered [ 2241.701711] Key type .llcrypt registered [ 2243.445556] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:03:33 (1786745013) [ 2244.149701] Key type .llcrypt unregistered [ 2244.154228] Key type ._llcrypt unregistered [ 2251.577726] Key type ._llcrypt registered [ 2251.584868] Key type .llcrypt registered [ 2251.785600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2251.848097] alg: No test for adler32 (adler32-zlib) [ 2259.775531] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:03:50 (1786745030) [ 2259.914349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.994728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.091745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.109748] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2260.956867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2261.000782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2261.031616] LNet: Removed LNI 192.168.202.42@tcp [ 2261.410191] LNet: 105737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.972593] Key type .llcrypt unregistered [ 2261.975404] Key type ._llcrypt unregistered [ 2274.868583] Key type ._llcrypt registered [ 2274.871581] Key type .llcrypt registered [ 2275.977116] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:04:07 (1786745047) [ 2276.388344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.399493] alg: No test for adler32 (adler32-zlib) [ 2277.322470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.378725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.400384] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2278.128201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2278.183149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.42@tcp [ 2278.203861] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2278.213988] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2278.216843] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2278, 0, 0 [ 2278.223385] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2278.228661] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2278.234396] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2278.238684] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2278, 0, 0 [ 2278.249577] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2278.256160] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2278.262505] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2278.266648] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2278, 0, 2328 [ 2278.276402] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2278.282296] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2278.290900] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2278.294341] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2278, 0, 2328 [ 2278.302363] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2285.676777] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:04:16 (1786745056) [ 2286.341543] LNet: 107277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.351205] LNetError: 107277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.388703] LNet: Removed LNI 192.168.202.42@tcp [ 2287.328516] Key type .llcrypt unregistered [ 2287.333237] Key type ._llcrypt unregistered [ 2293.910758] Key type ._llcrypt registered [ 2293.913854] Key type .llcrypt registered [ 2294.114220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2294.139222] alg: No test for adler32 (adler32-zlib) [ 2295.076958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2295.107922] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2302.637224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2309.617419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.643693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2309.662526] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 2310.551958] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2310.710641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2310.718246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2310.726815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2313.227987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2313.250152] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2322.619648] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:04:53 (1786745093) [ 2322.752524] LNet: 108778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.762649] LNetError: 108778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.773668] LNet: Removed LNI 192.168.202.42@tcp [ 2323.807462] LNet: Removed LNI 192.168.202.42@tcp1 [ 2323.811363] LNetError: 108778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2323.823787] LNetError: 108778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2323.843674] LNet: Removed LNI 10.1.2.3@tcp99 [ 2324.550336] Key type .llcrypt unregistered [ 2324.552827] Key type ._llcrypt unregistered [ 2330.428517] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2330.439948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2332.056724] Key type ._llcrypt registered [ 2332.061234] Key type .llcrypt registered [ 2332.373001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2332.396906] alg: No test for adler32 (adler32-zlib) [ 2333.369727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.431526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.468448] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2334.232869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2334.252791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2336.599508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2336.653744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2336.696341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2336.720805] LNetError: 109518:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2336.765380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2336.811715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2337.411471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2337.522247] LNet: 109539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.536093] LNetError: 109539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2337.544881] LNet: Removed LNI 10.1.2.3@tcp [ 2338.596958] LNet: Removed LNI 192.168.202.42@tcp [ 2339.404447] Key type .llcrypt unregistered [ 2339.406257] Key type ._llcrypt unregistered [ 2350.677606] Key type ._llcrypt registered [ 2350.684712] Key type .llcrypt registered [ 2351.959496] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:05:22 (1786745122) [ 2352.280296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.299301] alg: No test for adler32 (adler32-zlib) [ 2353.237887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.317409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.344817] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2354.154964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2354.244808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg242-client@tcp [ 2360.846607] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:05:31 (1786745131) [ 2361.141460] LNet: 111072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.147037] LNetError: 111072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.213470] LNet: Removed LNI 192.168.202.42@tcp [ 2362.941703] Key type .llcrypt unregistered [ 2362.944866] Key type ._llcrypt unregistered [ 2369.316405] Key type ._llcrypt registered [ 2369.317778] Key type .llcrypt registered [ 2369.531736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2369.539072] alg: No test for adler32 (adler32-zlib) [ 2370.441140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2370.479795] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2370.513336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2376.840778] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:05:47 (1786745147) [ 2378.499130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2380.354620] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:05:51 (1786745151) [ 2382.391206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2384.412756] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:05:55 (1786745155) [ 2384.616641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2384.632555] LNet: 112494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2384.642993] LNetError: 112494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.658797] LNet: Removed LNI 192.168.202.42@tcp [ 2384.672307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.709619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2384.734702] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2385.490717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2385.534194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2392.888873] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:06:04 (1786745164) [ 2393.198465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.221206] LNet: 113204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.226705] LNetError: 113204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.278510] LNet: Removed LNI 192.168.202.42@tcp [ 2394.324342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.402425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2394.433366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2394.764162] LNet: 113240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.229805] Key type .llcrypt unregistered [ 2395.233310] Key type ._llcrypt unregistered [ 2407.624481] Key type ._llcrypt registered [ 2407.626877] Key type .llcrypt registered [ 2408.999075] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:06:19 (1786745179) [ 2409.637036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.681409] alg: No test for adler32 (adler32-zlib) [ 2410.656505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.723714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.757985] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2411.529771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2411.576659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2417.360126] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:06:28 (1786745188) [ 2417.676439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.706769] LNet: 114759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.713311] LNetError: 114759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.727173] LNet: Removed LNI 192.168.202.42@tcp [ 2417.750781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.837892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2417.886474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2424.222868] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:06:35 (1786745195) [ 2424.329104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.365662] LNet: 115334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.400057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.490826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2424.849426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2424.925379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2425.025209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2425.502880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2425.544315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2425.624306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2425.646983] LNet: 115373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.673488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2425.737777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2426.097246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2426.130923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2426.209445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2426.618383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2426.654827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2433.250981] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:06:43 (1786745203) [ 2433.443634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.482449] LNet: 115968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.530830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.637733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2433.663233] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2434.667202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2434.691696] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2435.815467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2435.860724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2436.083208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.105971] LNetError: 116253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.112178] LNet: Removed LNI 192.168.202.42@tcp [ 2436.125251] LNet: Removed LNI 192.168.202.42@tcp2 [ 2436.142237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.202197] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2436.208986] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2436.215464] LNet: 116259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2436.807824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2437.106400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.132808] LNetError: 116296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.136074] LNetError: 116296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2437.152946] LNet: Removed LNI 192.168.202.42@tcp [ 2437.182974] LNet: Removed LNI 192.168.202.42@tcp2 [ 2437.201562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.254298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.269941] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2437.764492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2437.782765] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2438.682502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2438.736221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2438.828694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2439.147433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2439.165649] LNet: 116580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.172460] LNet: 116580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2439.179807] LNetError: 116580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.184291] LNetError: 116580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2439.200779] LNet: Removed LNI 192.168.202.42@tcp [ 2440.244548] LNet: Removed LNI 192.168.202.42@tcp2 [ 2440.310950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.391070] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2440.416918] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2440.432460] LNet: 116586:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2441.340319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2441.592540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.607677] LNet: Removed LNI 192.168.202.42@tcp [ 2442.669345] LNet: Removed LNI 192.168.202.42@tcp2 [ 2442.733618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.856494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.883449] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2443.633859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2443.667517] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2444.867392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2444.942727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2445.244428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.277663] LNetError: 116903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.288299] LNetError: 116903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2445.295517] LNet: Removed LNI 192.168.202.42@tcp [ 2445.308431] LNet: Removed LNI 192.168.202.42@tcp2 [ 2445.340585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.417848] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2445.428567] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2445.434759] LNet: 116909:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.167743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2446.425528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.519148] LNet: Removed LNI 192.168.202.42@tcp [ 2447.540748] LNet: Removed LNI 192.168.202.42@tcp2 [ 2447.602326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.701458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2447.748820] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2448.420733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2448.443161] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2449.226285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2449.262896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2449.326456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2449.585520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2449.603982] LNet: 117231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.608801] LNet: 117231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2450.656145] LNet: Removed LNI 192.168.202.42@tcp [ 2451.752414] LNet: Removed LNI 192.168.202.42@tcp2 [ 2451.845752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.936779] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2451.951750] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2451.957854] LNet: 117237:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.687556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2452.906608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2452.926750] LNet: Removed LNI 192.168.202.42@tcp [ 2453.991269] LNet: Removed LNI 192.168.202.42@tcp2 [ 2454.025585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.098015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2454.134483] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2455.140447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2455.168978] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2455.993043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2456.035778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2456.124925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2456.455700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.480457] LNetError: 117570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.485153] LNetError: 117570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2456.491917] LNet: Removed LNI 192.168.202.42@tcp [ 2456.511967] LNet: Removed LNI 192.168.202.42@tcp2 [ 2456.539694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.604657] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2456.610866] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2456.617755] LNet: 117577:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2457.224206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2457.633140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.719669] LNet: Removed LNI 192.168.202.42@tcp [ 2458.741263] LNet: Removed LNI 192.168.202.42@tcp2 [ 2458.796892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.918726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2458.982595] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2459.552797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2459.572469] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2460.416697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2460.452788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2460.487763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2460.520874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2460.911916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2461.983212] LNet: Removed LNI 192.168.202.42@tcp [ 2463.023329] LNet: Removed LNI 192.168.202.42@tcp2 [ 2463.062395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.173733] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2463.181986] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2463.773580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2470.528891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2472.700091] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:07:23 (1786745243) [ 2473.061424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.105454] LNet: 118696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.111583] LNet: 118696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2473.125816] LNetError: 118696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.133482] LNetError: 118696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2473.142592] LNet: Removed LNI 192.168.202.42@tcp [ 2474.228261] LNet: Removed LNI 192.168.202.42@tcp2 [ 2474.297743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.654897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.678930] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2475.689510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2475.762243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2476.613663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.631778] LNet: Removed LNI 10.1.2.3@tcp [ 2476.643739] LNet: Removed LNI 192.168.202.42@tcp [ 2476.660325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.742264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2476.775670] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2476.782490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2477.250262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2477.277567] LNet: Removed LNI 192.168.202.42@tcp [ 2477.293868] LNet: Removed LNI 10.1.2.3@tcp [ 2485.977958] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:07:36 (1786745256) [ 2486.174180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2486.226191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.309633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2486.426394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2486.432599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2486.441644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2487.893503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2487.946421] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2488.797806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2488.819966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2491.338641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2491.390304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2491.482387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.42@tcp -m PUT -e local_timeout -r 1 [ 2491.592459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2491.742058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2491.762604] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2492.146689] LNetError: 119625:0:(lib-move.c:4771:lnet_parse()) 192.168.202.42@tcp, src 192.168.202.42@tcp: Dropping PUT (error -108 looking up sender) [ 2493.152093] LNet: Removed LNI 10.1.2.3@tcp [ 2494.181652] LNet: Removed LNI 192.168.202.42@tcp [ 2494.745710] Key type .llcrypt unregistered [ 2494.748708] Key type ._llcrypt unregistered [ 2508.856323] Key type ._llcrypt registered [ 2508.860698] Key type .llcrypt registered [ 2510.919130] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:08:00 (1786745280) [ 2511.622217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.639951] alg: No test for adler32 (adler32-zlib) [ 2512.671141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.209870] LNet: 120938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.810286] Key type .llcrypt unregistered [ 2513.815764] Key type ._llcrypt unregistered [ 2521.770407] Key type ._llcrypt registered [ 2521.773187] Key type .llcrypt registered [ 2522.255383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2522.285732] alg: No test for adler32 (adler32-zlib) [ 2523.314860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2523.376342] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2530.290715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2530.564569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2531.232942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2531.522932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2531.558257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2531.589703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2531.617488] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 2531.642829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2531.700832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2531.743811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2531.777360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2531.816707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2531.835565] LNet: 121388:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2531.851298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2531.882472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2531.905623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2531.940481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2531.994576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2532.034892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2532.793504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2532.965628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.990983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2533.019579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2533.048551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2533.077806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2533.101404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2533.130953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2533.158973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2533.172965] LNet: 121505:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2533.188964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2533.224909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2533.260903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2533.330308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2533.399171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2533.443861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.284765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2534.512844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2534.541139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.587645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.616232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.649928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.673417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.703077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.742853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.768243] LNet: 121622:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.808246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.869456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.920178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.974897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2535.025520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2535.072724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.853417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2536.138930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2536.177678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2536.219462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2536.269794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2536.297458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2536.324671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2536.358772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2536.383053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2536.411561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2536.445976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2536.474079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2536.511345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2536.547097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2536.612760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2536.985975] LNet: 121786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.991906] LNetError: 121786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.012797] LNet: Removed LNI 192.168.202.42@tcp [ 2537.433499] Key type .llcrypt unregistered [ 2537.436329] Key type ._llcrypt unregistered [ 2554.699901] Key type ._llcrypt registered [ 2554.702616] Key type .llcrypt registered [ 2556.533946] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:08:46 (1786745326) [ 2556.845731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.861642] alg: No test for adler32 (adler32-zlib) [ 2557.890393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2558.434934] LNet: 122672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.452428] Key type .llcrypt unregistered [ 2559.457745] Key type ._llcrypt unregistered [ 2568.518148] Key type ._llcrypt registered [ 2568.529918] Key type .llcrypt registered [ 2569.004366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2569.032724] alg: No test for adler32 (adler32-zlib) [ 2570.219608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2570.246879] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2582.438380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2592.030117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2592.071744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2592.095788] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 2593.242703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2593.805876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2594.181585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2594.238523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2594.289248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2594.346512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2594.409441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2594.460477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2594.507449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2594.544733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2594.572642] LNet: 123326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2594.611549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2594.641730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2594.678574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2594.712473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2594.753534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2594.783684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2595.506342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2595.807490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2595.858787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2595.871167] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.142@tcp, 192.168.202.142@tcp1 [ 2595.909504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2595.963836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2596.003249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2596.029523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2596.055744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2596.085456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2596.102913] LNet: 123443:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2596.125796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2596.161244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2596.203447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2596.234748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2596.266319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2596.316577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2597.047409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2597.272902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2597.297869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2597.327647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2597.362393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2597.401339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2597.451517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2597.501486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2597.547680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2597.568179] LNet: 123560:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2597.593396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2597.623722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2597.663666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2597.698626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2597.735559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2597.775815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2598.461430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2598.756916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2598.806771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2598.836946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2598.892188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2598.940335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2599.002149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2599.066364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2599.131521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2599.178650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2599.243246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2599.303679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2599.347689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2599.431201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2599.487823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2605.581859] LNet: 123759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.586843] LNetError: 123759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.598535] LNet: Removed LNI 192.168.202.42@tcp [ 2605.613882] LNet: Removed LNI 192.168.202.42@tcp1 [ 2606.130752] Key type .llcrypt unregistered [ 2606.134117] Key type ._llcrypt unregistered [ 2624.089292] Key type ._llcrypt registered [ 2624.091340] Key type .llcrypt registered [ 2625.406602] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:09:56 (1786745396) [ 2626.930594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2628.828565] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:09:59 (1786745399) [ 2637.073646] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:10:07 (1786745407) [ 2644.619447] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:10:15 (1786745415) [ 2653.034259] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:10:24 (1786745424) [ 2660.932825] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:10:32 (1786745432) [ 2669.333784] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:10:40 (1786745440) [ 2676.550372] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:10:47 (1786745447) [ 2683.936526] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:10:54 (1786745454) [ 2691.726459] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:11:02 (1786745462) [ 2698.933687] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:11:10 (1786745470) [ 2706.032970] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:11:16 (1786745476) [ 2713.185470] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:11:24 (1786745484) [ 2720.065405] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:11:31 (1786745491) [ 2726.375645] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:11:37 (1786745497) [ 2733.604776] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:11:44 (1786745504) [ 2740.337898] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:11:51 (1786745511) [ 2748.625578] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:11:59 (1786745519) [ 2759.051620] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:12:09 (1786745529) [ 2759.714449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.763276] alg: No test for adler32 (adler32-zlib) [ 2760.780337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2760.993290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2761.034260] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2762.718834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2762.749309] LNet: 134759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.759863] LNetError: 134759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.788615] LNet: Removed LNI 192.168.202.42@tcp [ 2762.824708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2762.901224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2762.919822] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2771.220823] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:12:22 (1786745542) [ 2778.328263] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:12:29 (1786745549) [ 2786.120624] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:12:37 (1786745557) [ 2793.727057] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:12:44 (1786745564) [ 2803.245432] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:12:53 (1786745573) [ 2813.923405] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:13:04 (1786745584) [ 2822.796210] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:13:13 (1786745593) [ 2830.184493] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:13:20 (1786745600) [ 2836.927139] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:13:28 (1786745608) [ 2844.535618] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:13:35 (1786745615) [ 2850.768165] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:13:41 (1786745621) [ 2857.615315] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:13:48 (1786745628) [ 2865.649535] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2620 sec ========== 18:13:56 (1786745636) [ 2866.158530] LNet: 142224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.162516] LNetError: 142224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.178541] LNet: Removed LNI 192.168.202.42@tcp [ 2866.603872] Key type .llcrypt unregistered [ 2866.605508] Key type ._llcrypt unregistered [ 2882.312960] Key type ._llcrypt registered [ 2882.323513] Key type .llcrypt registered [ 2883.192823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2883.212459] alg: No test for adler32 (adler32-zlib) [ 2884.346807] Lustre: Lustre: Build Version: 2.17.55_1_g696fb08 [ 2884.697883] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2886.367170] Key type lgssc registered [ 2887.341438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.167778] Lustre: Mounted lustre-client [ 2948.230785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2956.655167] Lustre: Unmounted lustre-client [ 2995.222884] Key type lgssc unregistered [ 2995.469649] LNet: 145155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.474090] LNetError: 145155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2995.496648] LNet: Removed LNI 192.168.202.42@tcp [ 2996.226251] Key type .llcrypt unregistered [ 2996.227906] Key type ._llcrypt unregistered