[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455381221 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002370] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009126] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.020788] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024484] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] *** VALIDATE tmpfs *** [ 0.028424] *** VALIDATE proc *** [ 0.029196] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.032232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039086] debug: unmapping init [mem 0xffffffffab459000-0xffffffffab460fff] [ 0.041899] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042646] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.050265] rcu: Hierarchical SRCU implementation. [ 0.052324] smp: Bringing up secondary CPUs ... [ 0.053463] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057598] smp: Brought up 1 node, 4 CPUs [ 0.059010] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137144] node 0 deferred pages initialised in 75ms [ 0.140144] devtmpfs: initialized [ 0.141143] x86/mm: Memory block size: 128MB [ 0.143619] gcov: version magic: 0x41383552 [ 0.144585] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147236] pinctrl core: initialized pinctrl subsystem [ 0.148180] [ 0.148563] ************************************************************* [ 0.149010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ** ** [ 0.151009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152008] ** ** [ 0.153014] ** This means that this kernel is built to expose internal ** [ 0.154016] ** IOMMU data structures, which may compromise security on ** [ 0.155014] ** your system. ** [ 0.156009] ** ** [ 0.157011] ** If you see this message and you are not debugging the ** [ 0.158014] ** kernel, report this immediately to your vendor! ** [ 0.159011] ** ** [ 0.160010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161006] ************************************************************* [ 0.162617] NET: Registered protocol family 16 [ 0.163419] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167065] cpuidle: using governor menu [ 0.168809] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172784] PCI: Using configuration type 1 for base access [ 0.175113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185361] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192072] cryptd: max_cpu_qlen set to 1000 [ 0.195274] ACPI: Added _OSI(Module Device) [ 0.196025] ACPI: Added _OSI(Processor Device) [ 0.197015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199033] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205882] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210574] ACPI: Interpreter enabled [ 0.212064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214014] ACPI: Using IOAPIC for interrupt routing [ 0.215117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229438] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241400] acpiphp: Slot [2] registered [ 0.242100] acpiphp: Slot [5] registered [ 0.244107] acpiphp: Slot [6] registered [ 0.245082] acpiphp: Slot [3] registered [ 0.246114] acpiphp: Slot [4] registered [ 0.247061] acpiphp: Slot [7] registered [ 0.248067] acpiphp: Slot [8] registered [ 0.249085] acpiphp: Slot [9] registered [ 0.251099] acpiphp: Slot [10] registered [ 0.252075] acpiphp: Slot [11] registered [ 0.253105] acpiphp: Slot [12] registered [ 0.255073] acpiphp: Slot [13] registered [ 0.256096] acpiphp: Slot [14] registered [ 0.258147] acpiphp: Slot [15] registered [ 0.259110] acpiphp: Slot [16] registered [ 0.261069] acpiphp: Slot [17] registered [ 0.262076] acpiphp: Slot [18] registered [ 0.264100] acpiphp: Slot [19] registered [ 0.265099] acpiphp: Slot [20] registered [ 0.266085] acpiphp: Slot [21] registered [ 0.267085] acpiphp: Slot [22] registered [ 0.269055] acpiphp: Slot [23] registered [ 0.270099] acpiphp: Slot [24] registered [ 0.271111] acpiphp: Slot [25] registered [ 0.273093] acpiphp: Slot [26] registered [ 0.275171] acpiphp: Slot [27] registered [ 0.277136] acpiphp: Slot [28] registered [ 0.279157] acpiphp: Slot [29] registered [ 0.281186] acpiphp: Slot [30] registered [ 0.283149] acpiphp: Slot [31] registered [ 0.284089] PCI host bridge to bus 0000:00 [ 0.286037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303131] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305968] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309231] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316448] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.320040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.329016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331710] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334934] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338097] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.341442] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.347016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.358023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366275] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370859] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392008] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401238] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439232] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.441446] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.443375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.445364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.448243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.452214] iommu: Default domain type: Passthrough [ 0.455396] SCSI subsystem initialized [ 0.456150] ACPI: bus type USB registered [ 0.457102] usbcore: registered new interface driver usbfs [ 0.458117] usbcore: registered new interface driver hub [ 0.460074] usbcore: registered new device driver usb [ 0.462197] pps_core: LinuxPPS API ver. 1 registered [ 0.465014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.467048] PTP clock support registered [ 0.469046] EDAC MC: Ver: 3.0.0 [ 0.470446] PCI: Using ACPI for IRQ routing [ 0.472797] NetLabel: Initializing [ 0.474018] NetLabel: domain hash size = 128 [ 0.476018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.478127] NetLabel: unlabeled traffic allowed by default [ 0.480262] vgaarb: loaded [ 0.482290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.484016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.490074] clocksource: Switched to clocksource kvm-clock [ 0.600754] VFS: Disk quotas dquot_6.6.0 [ 0.602416] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.604752] *** VALIDATE ramfs *** [ 0.606539] *** VALIDATE hugetlbfs *** [ 0.608214] pnp: PnP ACPI init [ 0.609920] pnp: PnP ACPI: found 6 devices [ 0.632714] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.636564] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.638972] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.641318] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.644247] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.646291] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.648309] NET: Registered protocol family 2 [ 0.650687] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.655147] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.658656] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.663649] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.667364] TCP: Hash tables configured (established 65536 bind 65536) [ 0.670249] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.672950] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.675552] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.678132] NET: Registered protocol family 1 [ 0.680983] RPC: Registered named UNIX socket transport module. [ 0.683416] RPC: Registered udp transport module. [ 0.685246] RPC: Registered tcp transport module. [ 0.686962] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.689541] NET: Registered protocol family 44 [ 0.691639] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.693529] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.695793] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.697511] PCI: CLS 0 bytes, default 64 [ 0.699227] Unpacking initramfs... [ 2.089908] debug: unmapping init [mem 0xffff99a37cc64000-0xffff99a37ffcffff] [ 2.094050] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.096509] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.099694] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.600312] Initialise system trusted keyrings [ 2.602357] Key type blacklist registered [ 2.604694] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.616929] zbud: loaded [ 2.621522] *** VALIDATE nfs *** [ 2.623042] *** VALIDATE nfs4 *** [ 2.626082] pstore: using deflate compression [ 2.629465] Platform Keyring initialized [ 2.744489] NET: Registered protocol family 38 [ 2.746312] Key type asymmetric registered [ 2.747875] Asymmetric key parser 'x509' registered [ 2.750117] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.753619] io scheduler mq-deadline registered [ 2.755770] io scheduler kyber registered [ 2.757879] io scheduler bfq registered [ 2.760260] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.763824] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.768732] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.772351] ACPI: Power Button [PWRF] [ 2.780390] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.787345] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.800740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.829927] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.861062] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.866897] Non-volatile memory driver v1.3 [ 2.868406] Linux agpgart interface v0.103 [ 2.901991] virtio_blk virtio1: [vda] 150016 512-byte logical blocks (76.8 MB/73.3 MiB) [ 2.906421] vda: detected capacity change from 0 to 76808192 [ 2.924748] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.928290] vdb: detected capacity change from 0 to 1073741824 [ 2.946193] libphy: Fixed MDIO Bus: probed [ 2.959845] usbcore: registered new interface driver usbserial_generic [ 2.962804] usbserial: USB Serial support registered for generic [ 2.964984] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.969510] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.971275] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.973262] mousedev: PS/2 mouse device common for all mice [ 2.977953] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.978641] rtc_cmos 00:05: RTC can wake from S4 [ 2.985182] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.989721] rtc_cmos 00:05: registered as rtc0 [ 2.991206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.992599] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.998646] intel_pstate: CPU model not supported [ 3.002346] hid: raw HID events driver (C) Jiri Kosina [ 3.005495] usbcore: registered new interface driver usbhid [ 3.007545] usbhid: USB HID core driver [ 3.009238] drop_monitor: Initializing network drop monitor service [ 3.011962] Initializing XFRM netlink socket [ 3.014288] NET: Registered protocol family 10 [ 3.017445] Segment Routing with IPv6 [ 3.018695] NET: Registered protocol family 17 [ 3.020954] mpls_gso: MPLS GSO support [ 3.027437] RAS: Correctable Errors collector initialized. [ 3.029836] AVX version of gcm_enc/dec engaged. [ 3.031746] AES CTR mode by8 optimization enabled [ 3.109212] sched_clock: Marking stable (3109155719, 0)->(4001008645, -891852926) [ 3.113395] registered taskstats version 1 [ 3.115331] Loading compiled-in X.509 certificates [ 3.117715] zswap: loaded using pool lzo/zbud [ 3.144125] Key type big_key registered [ 3.156849] Key type encrypted registered [ 3.158544] ima: No TPM chip found, activating TPM-bypass! [ 3.160472] ima: Allocated hash algorithm: sha1 [ 3.162201] ima: No architecture policies found [ 3.163868] evm: Initialising EVM extended attributes: [ 3.165297] evm: security.selinux [ 3.166161] evm: security.ima [ 3.166824] evm: security.capability [ 3.167751] evm: HMAC attrs: 0x1 [ 3.170158] rtc_cmos 00:05: setting system clock to 2026-09-12 03:49:52 UTC (1789184992) [ 3.176386] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 3.179715] debug: unmapping init [mem 0xffffffffab182000-0xffffffffab458fff] [ 3.189134] Write protecting the kernel read-only data: 28672k [ 3.193195] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 3.196381] debug: unmapping init [mem 0xffffffffaa114000-0xffffffffaa1fffff] [ 3.230427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.238976] systemd[1]: Detected virtualization kvm. [ 3.241256] systemd[1]: Detected architecture x86-64. [ 3.243572] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.268439] systemd[1]: No hostname configured. [ 3.271506] systemd[1]: Set hostname to . [ 3.274051] random: systemd: uninitialized urandom read (16 bytes read) [ 3.277029] systemd[1]: Initializing machine ID from random generator. [ 3.332467] random: ln: uninitialized urandom read (6 bytes read) [ 3.421256] random: systemd: uninitialized urandom read (16 bytes read) [ 3.424336] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.431835] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.438071] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.070899] device-mapper: uevent: version 1.0.3 [ 4.073154] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.832209] virtio_net virtio0 ens2: renamed from eth0 [ 4.891795] scsi host0: ata_piix [ 4.902460] scsi host1: ata_piix [ 4.905290] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.910297] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.141500] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.144177] random: fast init done [ 9.680901] random: crng init done [ 9.682104] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.008361] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.064184] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.321809] SELinux: Disabled at runtime. [ 11.382925] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.390449] systemd[1]: Detected virtualization kvm. [ 11.392164] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.854484] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.857395] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.861884] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.866192] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.869144] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.874857] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.878683] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ 11.980312] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.211859] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.508261] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.526468] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.684904] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.694530] EDAC sbridge: Ver: 1.1.2 [ 13.585856] Key type dns_resolver registered [ 13.896377] NFS: Registering the id_resolver key type [ 13.898045] Key type id_resolver registered [ 13.899709] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg347-client login: [ 40.732648] libcfs: loading out-of-tree module taints kernel. [ 40.769679] Key type ._llcrypt registered [ 40.774277] Key type .llcrypt registered [ 41.070716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.077385] alg: No test for adler32 (adler32-zlib) [ 42.092378] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 42.454481] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 44.103352] Key type lgssc registered [ 44.847472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.973872] hrtimer: interrupt took 2996427 ns [ 160.366746] Lustre: Mounted lustre-client - version 2.17.58_40_g357d090 [ 165.023784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 175.701734] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing check_logdir /tmp/testlogs/ [ 179.937449] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing yml_node [ 183.489488] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 185.686449] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 185.825658] Lustre: lustre-OST0000-osc-ffff99a3d00ed800: disconnect after 24s idle [ 187.867742] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 189.239646] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 11 23:52:57 EDT 2026 [ 203.043553] Lustre: Unmounted lustre-client [ 219.820759] Key type lgssc unregistered [ 220.082850] LNet: 8396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 220.088941] LNetError: 8396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 220.120266] LNet: Removed LNI 192.168.203.47@tcp [ 220.908269] Key type .llcrypt unregistered [ 220.909904] Key type ._llcrypt unregistered [ 245.914901] Key type ._llcrypt registered [ 245.917739] Key type .llcrypt registered [ 246.106061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 246.130807] alg: No test for adler32 (adler32-zlib) [ 247.093970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 247.179769] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 247.212149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 251.869848] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 251.993996] LNet: 9268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 251.999985] LNetError: 9268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 252.025064] LNet: Removed LNI 192.168.203.47@tcp [ 252.646626] Key type .llcrypt unregistered [ 252.648744] Key type ._llcrypt unregistered [ 260.350111] Key type ._llcrypt registered [ 260.351897] Key type .llcrypt registered [ 262.457113] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:54:09 (1789185249) [ 262.971924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 262.994911] alg: No test for adler32 (adler32-zlib) [ 263.916655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.008730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.030153] LNet: 9684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.051376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.122432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 264.140575] LNet: 9691:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 271.668938] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:54:19 (1789185259) [ 271.972776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.022133] LNet: 10269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.043912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.109845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 278.191084] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:54:26 (1789185266) [ 278.351289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 278.392690] LNet: 10853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.415394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.486206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 284.599439] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:54:32 (1789185272) [ 284.868726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.922047] LNet: 11438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.952939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.075951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 292.296427] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:54:39 (1789185279) [ 292.456819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.496932] LNet: 12017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.526864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.653292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 293.235993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 300.791191] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:54:48 (1789185288) [ 300.840456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.864895] LNet: 12607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.878981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.965444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 301.516476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 302.091911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 309.829323] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:54:57 (1789185297) [ 310.269437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.338653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.436072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 318.467606] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:55:06 (1789185306) [ 318.580535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.599496] LNet: 13795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.608512] LNet: 13795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 318.628651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.678810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 318.707431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 318.976952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 319.005854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 319.251629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 319.277620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 319.633569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 319.653040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 320.032610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 320.077393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 320.382568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 320.417462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 320.841250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 320.891277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 321.517283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 321.569678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 322.178774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 322.218856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 322.577121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 322.609237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 323.014702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 323.051328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 323.369471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 323.411244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 323.759364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 323.791809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 330.343479] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:55:18 (1789185318) [ 330.780884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.815876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.865796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 330.881879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 338.456721] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:55:26 (1789185326) [ 338.588587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.680604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.808565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 338.843200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 346.182927] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:55:34 (1789185334) [ 346.539373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.637918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.737606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 346.769983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 353.381061] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:55:41 (1789185341) [ 353.507424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.528917] LNet: 16235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.534640] LNet: 16235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 353.561759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.657376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 353.685916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 360.020664] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:55:48 (1789185348) [ 360.129533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.175977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.264600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 360.296828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 367.280747] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:55:55 (1789185355) [ 367.479905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.530726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.590376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 367.638968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 373.873250] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:56:01 (1789185361) [ 374.069523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.129618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.204192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.599719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 380.793552] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:56:08 (1789185368) [ 380.882160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.935544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.154873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 381.349847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.493194] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:56:15 (1789185375) [ 387.712848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.754593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.943364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.093598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 394.256654] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:56:22 (1789185382) [ 394.327744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.365400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.764195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.035542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 400.897852] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:56:29 (1789185389) [ 401.252359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.290033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.384495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.436638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 408.036693] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:56:35 (1789185395) [ 408.290095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.333696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.393251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.463381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.979567] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:56:43 (1789185403) [ 415.237593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.269385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.360276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 421.471527] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:56:49 (1789185409) [ 421.659891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.708140] LNet: 22267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.711611] LNet: 22267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 421.733916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.814312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 427.411195] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:56:55 (1789185415) [ 427.689234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.726666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.794305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 428.169188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 433.591103] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:57:01 (1789185421) [ 433.761645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.807250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.910436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 433.944840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 438.985772] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:57:07 (1789185427) [ 439.250222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.287519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.365402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 439.406040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 444.951959] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:57:13 (1789185433) [ 445.074113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.152675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.256550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 445.318497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 451.134943] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:57:19 (1789185439) [ 451.409442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.493517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.593317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 451.637493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 457.088857] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:57:25 (1789185445) [ 457.353449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.399536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.476392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 457.537901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 457.555531] LNetError: 25800:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 457.600925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 457.646869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 463.296182] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:57:31 (1789185451) [ 463.608294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.662907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.766402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tVJPjcZDEA [ 463.867460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tVJPjcZDEA [ 463.923190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.tVJPjcZDEA [ 469.814246] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:57:37 (1789185457) [ 470.041418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.071958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.144348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 470.191280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 470.263956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 476.139495] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:57:44 (1789185464) [ 476.416504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.472363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.526740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 476.546905] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 478.166736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 479.336847] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 481.223417] Key type lgssc registered [ 482.243553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 485.720451] LNetError: 27561:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.47@tcp99 has no available nets [ 488.043115] Key type lgssc unregistered [ 488.423749] LNetError: 28666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.451992] LNet: Removed LNI 192.168.203.47@tcp [ 490.033395] Key type .llcrypt unregistered [ 490.040084] Key type ._llcrypt unregistered [ 495.547917] Key type ._llcrypt registered [ 495.549957] Key type .llcrypt registered [ 496.795811] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:58:04 (1789185484) [ 497.217331] Key type .llcrypt unregistered [ 497.226348] Key type ._llcrypt unregistered [ 502.992922] Key type ._llcrypt registered [ 502.995606] Key type .llcrypt registered [ 503.139380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 503.161331] alg: No test for adler32 (adler32-zlib) [ 504.090534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.168913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 504.195826] LNet: 29573:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 504.210796] LNet: 29573:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 504.220914] LNet: 29573:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 504.234572] LNet: Message forwarding will be enabled in 2s [ 507.295170] LNet: Message forwarding enabled [ 507.699264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 507.716630] LNet: Message forwarding will stop in 2s to 6s [ 511.398757] LNet: Message forwarding disabled [ 512.174758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 512.198296] LNet: 29671:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 512.209663] LNet: 29671:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 512.222137] LNet: 29671:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 512.233337] LNet: Message forwarding will be enabled in 2s [ 512.385268] LNet: 29684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.915564] Key type .llcrypt unregistered [ 512.918547] Key type ._llcrypt unregistered [ 522.819504] Key type ._llcrypt registered [ 522.824027] Key type .llcrypt registered [ 523.913445] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:58:32 (1789185512) [ 524.254420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 524.266728] alg: No test for adler32 (adler32-zlib) [ 525.161283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.231151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 530.140891] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:58:38 (1789185518) [ 530.399873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.418950] LNet: 31084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.449705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.493480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 530.518962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 530.544890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 530.569440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 530.601549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 530.628455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 530.664213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 530.701864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 530.738787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 530.762149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 530.797648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 530.827823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 530.853472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 530.877575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 530.925946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 530.959715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 530.998129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 531.044826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 531.068369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 531.085623] LNetError: 31127:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 536.695692] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:58:44 (1789185524) [ 536.930027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.949185] LNet: 31698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.967767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.034563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.808773] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:58:51 (1789185531) [ 544.453503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg347-server [ 545.798436] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:58:54 (1789185534) [ 545.983622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.009323] LNet: 32628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.026613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.108683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 546.139816] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 546.931385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[48-52]@tcp [ 546.941883] LNet: 32761:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 546.994531] LNetError: 32632:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.48@tcp is being used as a gateway but routing feature is not turned on [ 547.022697] LNetError: 32632:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.48@tcp has gone from up to down [ 553.005191] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:59:01 (1789185541) [ 554.890173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg347-server [ 556.940282] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:59:04 (1789185544) [ 557.129729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.170622] LNet: 33696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.180056] LNetError: 33696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.246989] LNet: Removed LNI 192.168.203.47@tcp [ 558.276455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.371911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 558.418455] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 559.201490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[48-53/2]@tcp [ 559.238785] LNet: 33814:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 559.251814] LNet: 33814:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 559.289338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[48-53/2]@tcp [ 559.296317] LNetError: 33701:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.48@tcp is being used as a gateway but routing feature is not turned on [ 559.322713] LNetError: 33701:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 559.327335] LNetError: 33701:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.48@tcp has gone from up to down [ 559.333474] LNetError: 33701:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 566.166043] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:59:13 (1789185553) [ 566.387779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.432704] LNet: 34404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.453960] LNetError: 34404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.498272] LNet: Removed LNI 192.168.203.47@tcp [ 566.565154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.667633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 566.798650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.886313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.994765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.046844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.092572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.133685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 567.246841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.280151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.358697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.486695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.573607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.671465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 567.796900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.886948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.031162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.122629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.179303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.249987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 568.325578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.366247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.442814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.546633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.591260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.647535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 568.733544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.770755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.847371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.966481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.014147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.059640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 569.112870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.151582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.217406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 575.538272] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:59:23 (1789185563) [ 577.025393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg347-server [ 578.900632] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:59:26 (1789185566) [ 580.663561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg347-server [ 582.628187] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:59:30 (1789185570) [ 582.708934] LNet: 35799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.717661] LNet: 35799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 583.325198] Key type .llcrypt unregistered [ 583.329150] Key type ._llcrypt unregistered [ 589.792982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 589.943938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 589.955335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 589.963339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 592.439083] Key type ._llcrypt registered [ 592.440555] Key type .llcrypt registered [ 592.590796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.600786] alg: No test for adler32 (adler32-zlib) [ 593.487878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.530772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.551201] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 594.171514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 594.188120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 596.027813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 596.072358] LNet: Removed LNI 10.1.2.3@tcp [ 596.218938] LNet: 36521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.227844] LNetError: 36521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.247918] LNet: Removed LNI 192.168.203.47@tcp [ 596.818471] Key type .llcrypt unregistered [ 596.820083] Key type ._llcrypt unregistered [ 607.182712] Key type ._llcrypt registered [ 607.189058] Key type .llcrypt registered [ 608.449390] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:59:56 (1789185596) [ 609.064538] Key type .llcrypt unregistered [ 609.070762] Key type ._llcrypt unregistered [ 616.651297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 616.857438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 616.870373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 616.874513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 619.429427] Key type ._llcrypt registered [ 619.432335] Key type .llcrypt registered [ 619.596888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 619.609316] alg: No test for adler32 (adler32-zlib) [ 620.533229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.623947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 620.671388] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 621.532398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 621.553676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.722327] LNet: 38014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.729126] LNetError: 38014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 624.736465] LNet: Removed LNI 10.1.2.3@tcp [ 624.752638] LNet: Removed LNI 192.168.203.47@tcp [ 625.600282] Key type .llcrypt unregistered [ 625.606669] Key type ._llcrypt unregistered [ 637.492715] Key type ._llcrypt registered [ 637.495270] Key type .llcrypt registered [ 638.614634] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:00:26 (1789185626) [ 639.427505] Key type .llcrypt unregistered [ 639.430824] Key type ._llcrypt unregistered [ 645.199924] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 645.322367] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 645.325248] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 645.328535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 647.774451] Key type ._llcrypt registered [ 647.776569] Key type .llcrypt registered [ 647.929599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.941570] alg: No test for adler32 (adler32-zlib) [ 648.829551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.868472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 648.886588] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 650.237628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 650.255500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.148578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 652.171320] LNet: Removed LNI 10.1.2.3@tcp [ 652.203236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 652.253290] LNet: Removed LNI 10.1.2.31@tcp [ 652.283436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 652.299073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.927248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 652.959679] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 653.723466] LNet: 39928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.728662] LNetError: 39928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 653.734391] LNet: Removed LNI 10.1.2.31@tcp [ 653.746405] LNet: Removed LNI 10.1.2.3@tcp [ 654.265552] Key type .llcrypt unregistered [ 654.268804] Key type ._llcrypt unregistered [ 665.419718] Key type ._llcrypt registered [ 665.421873] Key type .llcrypt registered [ 666.694859] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:00:54 (1789185654) [ 667.088572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.104136] alg: No test for adler32 (adler32-zlib) [ 668.002300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.054373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.071598] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 668.687448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.704238] LNet: 40878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.711045] LNetError: 40878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.725367] LNet: Removed LNI 192.168.203.47@tcp [ 668.742212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.798740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.47@tcp [ 668.821326] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 669.552308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.47@tcp [ 669.586822] LNet: Removed LNI 192.168.203.47@tcp [ 669.689943] LNet: 41013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.384628] Key type .llcrypt unregistered [ 670.387496] Key type ._llcrypt unregistered [ 680.917949] Key type ._llcrypt registered [ 680.925426] Key type .llcrypt registered [ 682.047124] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:01:10 (1789185670) [ 683.783518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg347-server [ 685.403987] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:01:13 (1789185673) [ 685.811274] Key type .llcrypt unregistered [ 685.815634] Key type ._llcrypt unregistered [ 691.732813] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 691.984672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 691.993735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 691.998786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 694.533069] Key type ._llcrypt registered [ 694.535557] Key type .llcrypt registered [ 694.693279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 694.705797] alg: No test for adler32 (adler32-zlib) [ 695.628552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.707260] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 696.236068] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 697.924903] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.931731] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.937293] LNet: Removed LNI 192.168.203.47@tcp1 [ 697.947460] LNet: Removed LNI 10.1.2.3@tcp2 [ 698.461541] Key type .llcrypt unregistered [ 698.467260] Key type ._llcrypt unregistered [ 708.460945] Key type ._llcrypt registered [ 708.467866] Key type .llcrypt registered [ 709.633357] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:01:37 (1789185697) [ 710.156378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.168958] alg: No test for adler32 (adler32-zlib) [ 711.120719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.186595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.217421] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 712.075628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.114941] LNet: 43826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.127511] LNetError: 43826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.171472] LNet: Removed LNI 192.168.203.47@tcp [ 712.190588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.246912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 712.262624] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 718.575463] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:01:46 (1789185706) [ 718.947880] LNet: 44418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.954586] LNetError: 44418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.971285] LNet: Removed LNI 192.168.203.47@tcp [ 719.547448] Key type .llcrypt unregistered [ 719.550768] Key type ._llcrypt unregistered [ 724.333264] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.496056] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.501914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.506491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.031420] Key type ._llcrypt registered [ 727.033535] Key type .llcrypt registered [ 727.198329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.227798] alg: No test for adler32 (adler32-zlib) [ 728.143136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.199736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.223435] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 728.833459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 728.849886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.456883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.471452] LNet: 45156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.476449] LNetError: 45156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 730.481582] LNet: Removed LNI 10.1.2.3@tcp [ 730.490961] LNet: Removed LNI 192.168.203.47@tcp [ 730.508104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.559271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 730.590212] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 730.595821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.870503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.884613] LNet: Removed LNI 10.1.2.3@tcp [ 730.893370] LNet: Removed LNI 192.168.203.47@tcp [ 730.905480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.949832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 730.965130] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 730.968623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.241383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 731.257934] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 731.983700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.997407] LNet: 45345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.002729] LNet: 45345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 732.008524] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 732.013484] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 732.020107] LNet: Removed LNI 10.1.2.3@tcp [ 732.022565] LNet: Removed LNI 192.168.203.47@tcp [ 732.030413] LNet: Removed LNI 192.168.203.47@tcp2 [ 732.045948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.083774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 732.104024] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 732.106815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.110390] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 738.757395] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:02:06 (1789185726) [ 738.975164] LNet: 45941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.990400] LNetError: 45941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.004289] LNetError: 45941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 739.017650] LNet: Removed LNI 10.1.2.3@tcp [ 739.020760] LNet: Removed LNI 192.168.203.47@tcp [ 739.035334] LNet: Removed LNI 192.168.203.47@tcp2 [ 740.204442] Key type .llcrypt unregistered [ 740.210397] Key type ._llcrypt unregistered [ 747.545635] Key type ._llcrypt registered [ 747.547522] Key type .llcrypt registered [ 747.690718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.706062] alg: No test for adler32 (adler32-zlib) [ 748.617688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.716025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.747343] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 749.929635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.944883] LNet: 46335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.951152] LNetError: 46335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.964762] LNet: Removed LNI 192.168.203.47@tcp [ 749.982776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.080841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 750.121025] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 750.751728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.793061] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.818448] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.852471] LNet: Removed LNI 192.168.203.47@tcp [ 750.884876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.008915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 751.054349] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 751.854229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.925730] LNet: 46395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.932850] LNetError: 46395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.948955] LNet: Removed LNI 192.168.203.47@tcp [ 751.963831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.092567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 752.118707] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 753.097866] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.317721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.331732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.348834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.945410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.975377] LNet: 46475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.985980] LNetError: 46475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.033550] LNet: Removed LNI 192.168.203.47@tcp [ 756.086026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.161567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.193676] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 762.621459] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:02:30 (1789185750) [ 762.998434] LNet: 47074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.014503] LNetError: 47074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.041809] LNet: Removed LNI 192.168.203.47@tcp [ 763.769278] Key type .llcrypt unregistered [ 763.771059] Key type ._llcrypt unregistered [ 768.668720] Key type ._llcrypt registered [ 768.672808] Key type .llcrypt registered [ 768.882546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.894950] alg: No test for adler32 (adler32-zlib) [ 769.823560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.876781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.907615] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 770.640694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.654556] LNet: 47469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.666740] LNetError: 47469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.684616] LNet: Removed LNI 192.168.203.47@tcp [ 770.702644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.762264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 770.820232] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 775.915744] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:02:44 (1789185764) [ 776.019722] LNet: 48054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.038638] LNetError: 48054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.076723] LNet: Removed LNI 192.168.203.47@tcp [ 776.620562] Key type .llcrypt unregistered [ 776.624761] Key type ._llcrypt unregistered [ 782.995633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.183249] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.194365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.206286] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.127411] Key type ._llcrypt registered [ 786.135080] Key type .llcrypt registered [ 786.361556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.401148] alg: No test for adler32 (adler32-zlib) [ 787.294301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.345306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.378570] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 788.347450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.371586] LNet: 48509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.385190] LNetError: 48509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.408079] LNet: Removed LNI 192.168.203.47@tcp [ 788.426574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.528673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 788.578368] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 796.895358] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:03:04 (1789185784) [ 797.041304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.070505] LNet: 49102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.084613] LNetError: 49102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.121139] LNet: Removed LNI 192.168.203.47@tcp [ 797.163659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.298236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 802.792472] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:03:11 (1789185791) [ 803.253621] LNet: 49691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.874384] Key type .llcrypt unregistered [ 803.878769] Key type ._llcrypt unregistered [ 809.800774] Key type ._llcrypt registered [ 809.802714] Key type .llcrypt registered [ 810.062702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.074313] alg: No test for adler32 (adler32-zlib) [ 810.975762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.027416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.058606] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 811.901664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.929918] LNet: 50086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.936545] LNetError: 50086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.945909] LNet: Removed LNI 192.168.203.47@tcp [ 811.965478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.064401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 812.100326] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 819.227182] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:03:26 (1789185806) [ 819.375757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.417215] LNet: 50674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.427244] LNetError: 50674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.456920] LNet: Removed LNI 192.168.203.47@tcp [ 819.539441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.628352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 828.725614] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:03:36 (1789185816) [ 830.593539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 832.663738] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:03:40 (1789185820) [ 834.410585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 836.348640] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:03:44 (1789185824) [ 838.013605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 839.598926] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:03:47 (1789185827) [ 841.031915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 843.235381] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:03:51 (1789185831) [ 844.737873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 846.326274] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:03:54 (1789185834) [ 846.654761] LNet: 52704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.283546] Key type .llcrypt unregistered [ 847.289472] Key type ._llcrypt unregistered [ 852.890577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.994364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.999207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.008976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.662242] Key type ._llcrypt registered [ 855.665177] Key type .llcrypt registered [ 855.847530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.866814] alg: No test for adler32 (adler32-zlib) [ 856.796523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.883937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 856.900892] LNet: Added LNI 192.168.203.47@tcp [8/512/0/180] [ 856.907865] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 856.912653] LNet: 53022:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 856.995133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.009428] LNet: 53041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.015768] LNetError: 53041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 857.028073] LNet: Removed LNI 10.1.2.3@tcp [ 857.029642] LNetError: 53041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.059713] LNet: Removed LNI 192.168.203.47@tcp [ 857.083912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.157703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 857.184711] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 857.287841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.322492] LNet: 53065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.330896] LNetError: 53065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.347529] LNet: Removed LNI 192.168.203.47@tcp [ 857.380321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.433837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 857.453575] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 857.461181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.464209] LNet: 53073:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 857.535205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.560772] LNet: 53091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.567837] LNetError: 53091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 857.575226] LNet: Removed LNI 10.1.2.3@tcp [ 857.578310] LNetError: 53091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.601183] LNet: Removed LNI 192.168.203.47@tcp [ 857.619640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.702317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 857.747760] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 857.759284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.769496] LNet: 53099:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 858.007644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.024338] LNet: 53125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.028916] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.033668] LNet: Removed LNI 10.1.2.3@tcp [ 858.036316] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.075028] LNet: Removed LNI 192.168.203.47@tcp [ 858.118169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.205389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.256274] LNet: Added LNI 192.168.203.47@tcp [8/256/0/182] [ 858.265939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 858.401291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.445725] LNet: 53153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.457507] LNetError: 53153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.463257] LNet: Removed LNI 10.1.2.3@tcp [ 858.464672] LNetError: 53153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.494080] LNet: Removed LNI 192.168.203.47@tcp [ 858.548293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.618555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.642789] LNet: Added LNI 192.168.203.47@tcp [8/256/0/182] [ 858.647195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 858.650883] LNet: 53161:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 858.722176] LNet: Removed LNI 10.1.2.3@tcp [ 858.743706] LNet: Removed LNI 192.168.203.47@tcp [ 859.416812] Key type .llcrypt unregistered [ 859.422505] Key type ._llcrypt unregistered [ 869.791296] Key type ._llcrypt registered [ 869.792924] Key type .llcrypt registered [ 870.974529] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:04:18 (1789185858) [ 872.446015] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 874.030971] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:04:22 (1789185862) [ 874.394484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.412610] alg: No test for adler32 (adler32-zlib) [ 875.290220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.345921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.378785] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 876.009446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.47 [ 876.028976] LNet: 54422:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 876.034845] LNet: Removed LNI 192.168.203.47@tcp [ 881.190555] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:04:29 (1789185869) [ 881.355584] LNet: 54987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.363744] LNetError: 54987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.386603] LNet: Removed LNI 192.168.203.47@tcp [ 882.038708] Key type .llcrypt unregistered [ 882.042389] Key type ._llcrypt unregistered [ 887.662833] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 887.887792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.470282] Key type ._llcrypt registered [ 890.472356] Key type .llcrypt registered [ 890.590858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.621232] alg: No test for adler32 (adler32-zlib) [ 891.857375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.520354] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:04:45 (1789185885) [ 897.651834] LNet: 55896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.667426] LNetError: 55896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.691818] LNet: Removed LNI 10.1.2.3@tcp [ 898.373448] Key type .llcrypt unregistered [ 898.378315] Key type ._llcrypt unregistered [ 903.619133] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 903.873509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 903.875963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 903.882610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.524589] Key type ._llcrypt registered [ 906.529345] Key type .llcrypt registered [ 906.674871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.687475] alg: No test for adler32 (adler32-zlib) [ 907.600855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 912.903974] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:05:01 (1789185901) [ 914.087973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 915.647570] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:05:03 (1789185903) [ 915.791888] LNet: 57087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.797675] LNetError: 57087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 915.808817] LNet: Removed LNI 10.1.2.3@tcp [ 916.368456] Key type .llcrypt unregistered [ 916.371712] Key type ._llcrypt unregistered [ 921.625611] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 921.912553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 921.915508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 921.918756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 923.449308] Key type ._llcrypt registered [ 923.451549] Key type .llcrypt registered [ 923.598981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.611100] alg: No test for adler32 (adler32-zlib) [ 924.508149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.567914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.519899] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:05:20 (1789185920) [ 932.963297] LNet: 58306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.970253] LNetError: 58306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.990806] LNet: Removed LNI 10.1.2.3@tcp [ 933.510689] Key type .llcrypt unregistered [ 933.513764] Key type ._llcrypt unregistered [ 938.589538] Key type ._llcrypt registered [ 938.593962] Key type .llcrypt registered [ 938.821865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.845173] alg: No test for adler32 (adler32-zlib) [ 939.760674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 939.781145] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 946.959944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 947.657838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_interrupt -r 1 [ 947.690496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_interrupt -r 1 [ 947.730509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 947.748284] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 947.955643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 947.982623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 948.000568] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 948.012212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 948.046909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.078406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 948.105882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.136727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.166735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.191183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.204830] LNet: 58742:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 948.220917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.243536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.267066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.291874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.318185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.345756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.373323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 948.765458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_dropped -r 1 [ 948.811780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_dropped -r 1 [ 948.872303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 948.903886] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 949.172182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 949.203620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.232511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.260423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.284112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 949.309629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.331961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.357474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.379148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.391858] LNet: 58853:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.409519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.434379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 949.465918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 949.497535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 949.531622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.577499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.607525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 950.106402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_aborted -r 1 [ 950.135115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_aborted -r 1 [ 950.163661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 950.176045] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 950.178817] LNet: Skipped 1 previous similar message [ 950.358454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.384961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 950.408796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.445252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.483684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 950.522226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.549716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.578862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 950.621407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 950.637854] LNet: 58964:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 950.658863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 950.684672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.710244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.732964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 950.756918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.784833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 950.812585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 951.219304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_no_route -r 1 [ 951.263472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_no_route -r 1 [ 951.292825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 951.540881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 951.560715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 951.584595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 951.605099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 951.627819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 951.656820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 951.680623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 951.700762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.720547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.740921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.764466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.786439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.811209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.837359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.869379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.893480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 952.286685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_timeout -r 1 [ 952.317593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_timeout -r 1 [ 952.362481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 952.379419] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 952.385381] LNet: Skipped 1 previous similar message [ 952.615431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.645337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 952.675322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.702847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.733809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 952.761421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.788809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.839156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.884373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 952.901660] LNet: 59187:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.906209] LNet: 59187:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 952.919798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 952.940744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 952.963720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 952.985186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 953.011554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.038506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.063398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 953.614378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_error -r 1 [ 953.642723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_error -r 1 [ 953.685598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 953.897618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 953.922433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 953.945240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 953.978831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.003559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 954.039637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.086195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.131302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.165756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.187947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.227483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.261372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.300941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.335146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.357305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.395252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 954.626301] LNet: 59339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.631969] LNetError: 59339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.649251] LNet: Removed LNI 192.168.203.47@tcp [ 954.979773] Key type .llcrypt unregistered [ 954.982112] Key type ._llcrypt unregistered [ 970.799580] Key type ._llcrypt registered [ 970.801845] Key type .llcrypt registered [ 971.998648] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:05:59 (1789185959) [ 972.714488] Key type .llcrypt unregistered [ 972.718776] Key type ._llcrypt unregistered [ 979.262391] Key type ._llcrypt registered [ 979.266556] Key type .llcrypt registered [ 979.452881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.471615] alg: No test for adler32 (adler32-zlib) [ 980.421481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 980.450652] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 988.367207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 998.241075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.307184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 998.347513] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 999.865252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_interrupt -r 1 [ 999.898302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_interrupt -r 1 [ 999.948216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_interrupt -r 1 [ 999.979238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_interrupt -r 1 [ 1000.011578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_interrupt -r 1 [ 1000.040656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_interrupt -r 1 [ 1000.075275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_interrupt -r 1 [ 1000.130434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_interrupt -r 1 [ 1000.167206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1000.192995] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1000.199122] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1000.457557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.492474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.525765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.561190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.585955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1000.611089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.640534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.669306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.694994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.708476] LNet: 60813:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1000.726783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.772620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.819583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.865105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.903416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.938510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.966455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1001.538701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_dropped -r 1 [ 1001.566239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_dropped -r 1 [ 1001.592220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_dropped -r 1 [ 1001.640422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_dropped -r 1 [ 1001.677589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_dropped -r 1 [ 1001.720496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_dropped -r 1 [ 1001.773626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_dropped -r 1 [ 1001.816042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_dropped -r 1 [ 1001.845374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1001.872171] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1001.877548] LNet: Skipped 2 previous similar messages [ 1002.151266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.186901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.214191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.243741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.274292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1002.305173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.327563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.352756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.377618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.393546] LNet: 60936:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1002.414157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.443402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.482915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.534385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.573421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.610336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.662542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1003.311975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_aborted -r 1 [ 1003.352812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_aborted -r 1 [ 1003.396705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_aborted -r 1 [ 1003.437632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_aborted -r 1 [ 1003.486577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_aborted -r 1 [ 1003.545663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_aborted -r 1 [ 1003.579466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_aborted -r 1 [ 1003.613847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_aborted -r 1 [ 1003.648626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1003.670475] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1003.675940] LNet: Skipped 2 previous similar messages [ 1003.877380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.905417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1003.944226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.984654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.012594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1004.050636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.088893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.128426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1004.176389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1004.200797] LNet: 61060:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1004.233838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1004.263079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1004.286384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1004.331728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1004.370260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.422577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.456747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1004.876893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_no_route -r 1 [ 1004.911428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_no_route -r 1 [ 1004.942533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_no_route -r 1 [ 1004.982896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_no_route -r 1 [ 1005.015148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_no_route -r 1 [ 1005.039492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_no_route -r 1 [ 1005.067652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_no_route -r 1 [ 1005.091599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_no_route -r 1 [ 1005.118993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1005.316743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1005.346480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1005.375777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.396701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.420716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1005.442246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1005.464800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.485284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.517593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.554654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.591888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.626851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.669561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.702245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.745678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.770996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1006.308534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_timeout -r 1 [ 1006.347283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_timeout -r 1 [ 1006.393861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_timeout -r 1 [ 1006.422147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_timeout -r 1 [ 1006.464469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_timeout -r 1 [ 1006.500636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_timeout -r 1 [ 1006.533718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_timeout -r 1 [ 1006.571327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_timeout -r 1 [ 1006.620552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1006.647481] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1006.653276] LNet: Skipped 5 previous similar messages [ 1006.816356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.844444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.873901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.902112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.926936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1006.960150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.986980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1007.041856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1007.088883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1007.124333] LNet: 61306:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1007.134146] LNet: 61306:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1007.150305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.189609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.237265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.273829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.328279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.368595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.415845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1007.848400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_error -r 1 [ 1007.874794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_error -r 1 [ 1007.906845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_error -r 1 [ 1007.945128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_error -r 1 [ 1007.979540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_error -r 1 [ 1008.019342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_error -r 1 [ 1008.059785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_error -r 1 [ 1008.101324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_error -r 1 [ 1008.147905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1008.437305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1008.472711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1008.503454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.527813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.554745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1008.577613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1008.610814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.648798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.706989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.754164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.795152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.843969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.884994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.919207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.949478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.972767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.939199] LNet: 61506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.947455] LNetError: 61506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.955031] LNet: Removed LNI 192.168.203.47@tcp [ 1013.976135] LNet: Removed LNI 192.168.203.47@tcp1 [ 1014.374578] Key type .llcrypt unregistered [ 1014.380497] Key type ._llcrypt unregistered [ 1029.765426] Key type ._llcrypt registered [ 1029.770830] Key type .llcrypt registered [ 1031.001360] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:06:58 (1789186018) [ 1031.890453] Key type .llcrypt unregistered [ 1031.895309] Key type ._llcrypt unregistered [ 1037.461317] Key type ._llcrypt registered [ 1037.463223] Key type .llcrypt registered [ 1037.696939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.712161] alg: No test for adler32 (adler32-zlib) [ 1038.649162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1038.675638] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1045.395834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1046.200131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_dropped -r 1 [ 1046.244995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_dropped -r 1 [ 1046.314866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1046.336268] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1046.574533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.603885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.648954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.693874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.738710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1046.792590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.835879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.873427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.921760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.945772] LNet: 62753:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.980038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.003854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.036608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.086775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.122542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.164638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.202476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.682827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_error -r 1 [ 1047.713918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_error -r 1 [ 1047.764828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1047.785455] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1047.984754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.014713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.044960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.066403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.096679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1048.123191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.144753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.166477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.200657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.217838] LNet: 62864:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.240228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.276969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.306690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.341513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.390981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.455188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.500858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1048.996685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_timeout -r 1 [ 1049.028134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_timeout -r 1 [ 1049.061778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1049.081735] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1049.299103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.334105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1049.368993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.414493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.453600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1049.492944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.524946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.557411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.614934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.633278] LNet: 62976:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.649749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.675981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.707264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.744141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.781182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.817910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.851985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1050.155332] LNet: 63017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.163559] LNetError: 63017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.180501] LNet: Removed LNI 192.168.203.47@tcp [ 1050.505723] Key type .llcrypt unregistered [ 1050.507547] Key type ._llcrypt unregistered [ 1066.380964] Key type ._llcrypt registered [ 1066.383329] Key type .llcrypt registered [ 1067.582116] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:07:35 (1789186055) [ 1068.346398] Key type .llcrypt unregistered [ 1068.352179] Key type ._llcrypt unregistered [ 1074.258210] Key type ._llcrypt registered [ 1074.264213] Key type .llcrypt registered [ 1074.522831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.542547] alg: No test for adler32 (adler32-zlib) [ 1075.495524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1075.517621] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1081.560464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1088.525448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.557717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1088.584688] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1089.656171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_dropped -r 1 [ 1089.700664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e remote_dropped -r 1 [ 1089.737475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_dropped -r 1 [ 1089.783982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e remote_dropped -r 1 [ 1089.815863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e remote_dropped -r 1 [ 1089.842895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e remote_dropped -r 1 [ 1089.879723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e remote_dropped -r 1 [ 1089.916599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e remote_dropped -r 1 [ 1089.951491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1089.976359] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1089.982705] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.147@tcp [ 1090.131107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.155685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.182263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.209297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.235205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1090.261904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.288922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.322363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.361913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.388268] LNet: 64482:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.412912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.439234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.467185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.489479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.518810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.554266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.595236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1090.626190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.061585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_error -r 1 [ 1091.102255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e remote_error -r 1 [ 1091.136710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_error -r 1 [ 1091.165163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e remote_error -r 1 [ 1091.205759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e remote_error -r 1 [ 1091.247027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e remote_error -r 1 [ 1091.283202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e remote_error -r 1 [ 1091.323869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e remote_error -r 1 [ 1091.358534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1091.374333] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1091.379480] LNet: Skipped 2 previous similar messages [ 1091.615377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.642168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.663161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.693369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.729373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1091.769829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.798468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.824193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.858872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.883141] LNet: 64607:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.897923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.919987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.954270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.981246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.008525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.034908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.080361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1092.122605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.602571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_timeout -r 1 [ 1092.642277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e remote_timeout -r 1 [ 1092.674319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_timeout -r 1 [ 1092.701601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e remote_timeout -r 1 [ 1092.735331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e remote_timeout -r 1 [ 1092.767810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e remote_timeout -r 1 [ 1092.802583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e remote_timeout -r 1 [ 1092.836710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e remote_timeout -r 1 [ 1092.887179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.147@tcp [ 1092.913814] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1093.114285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.140385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.164740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.188908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.211477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1093.230858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.260252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.285534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.307706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.326556] LNet: 64732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.349372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.394882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.420404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.463693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.499834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.535868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.570823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.597182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.690989] LNet: 64810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.697345] LNetError: 64810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.700372] LNet: Removed LNI 192.168.203.47@tcp [ 1097.707769] LNet: Removed LNI 192.168.203.47@tcp1 [ 1098.053769] Key type .llcrypt unregistered [ 1098.058440] Key type ._llcrypt unregistered [ 1110.597765] Key type ._llcrypt registered [ 1110.601690] Key type .llcrypt registered [ 1111.625960] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:08:19 (1789186099) [ 1112.712327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1114.164715] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:08:22 (1789186102) [ 1114.861551] Key type .llcrypt unregistered [ 1114.864199] Key type ._llcrypt unregistered [ 1119.628375] Key type ._llcrypt registered [ 1119.633959] Key type .llcrypt registered [ 1119.811614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.827338] alg: No test for adler32 (adler32-zlib) [ 1120.720200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1120.758465] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1127.074876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1127.639709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e network_timeout -r 1 [ 1127.665344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e network_timeout -r 1 [ 1127.704303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1127.733923] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1127.945806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.967581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.975967] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1128.003408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.024030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.057172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1128.089968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.138438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.185088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.216698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.230812] LNet: 66365:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.246395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.268728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.288981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.312236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.331877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.351123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.585386] LNet: 66404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.595337] LNetError: 66404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.622092] LNet: Removed LNI 192.168.203.47@tcp [ 1128.879606] Key type .llcrypt unregistered [ 1128.883540] Key type ._llcrypt unregistered [ 1144.887569] Key type ._llcrypt registered [ 1144.899681] Key type .llcrypt registered [ 1145.120966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.141512] alg: No test for adler32 (adler32-zlib) [ 1146.053933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.087315] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1152.591851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1160.022402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.067584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1160.096589] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1161.342395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e network_timeout -r 1 [ 1161.385081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e network_timeout -r 1 [ 1161.429620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e network_timeout -r 1 [ 1161.473523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e network_timeout -r 1 [ 1161.516514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e network_timeout -r 1 [ 1161.552616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e network_timeout -r 1 [ 1161.588603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e network_timeout -r 1 [ 1161.635323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e network_timeout -r 1 [ 1161.682319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1161.714751] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1161.919143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.943815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.973498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.993102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.028645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1162.066198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.097155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.138878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.174875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.189277] LNet: 67281:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.209924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.252864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.285559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.307762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.331462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.357585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.880128] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1167.138317] LNet: 67355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.142552] LNetError: 67355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.150642] LNet: Removed LNI 192.168.203.47@tcp [ 1167.158053] LNet: Removed LNI 192.168.203.47@tcp1 [ 1167.523068] Key type .llcrypt unregistered [ 1167.528331] Key type ._llcrypt unregistered [ 1180.441493] Key type ._llcrypt registered [ 1180.443963] Key type .llcrypt registered [ 1181.739211] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:09:29 (1789186169) [ 1182.040598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.063483] alg: No test for adler32 (adler32-zlib) [ 1182.969955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1183.051937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1183.087303] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1183.711890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.737413] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1184.419402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1184.493444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1184.527870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1184.558359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1184.601286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1184.626720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1184.656497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.47@tcp [ 1184.675390] LNetError: 68454:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.47@tcp to recovery [ 1185.059286] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1185.063500] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1187.103589] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1188.052744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1191.199984] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1191.444259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1191.481248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.550802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1191.570965] LNet: 68561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.578895] LNetError: 68561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.608874] LNet: Removed LNI 192.168.203.47@tcp [ 1192.614473] LNetError: 68561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.631913] LNet: Removed LNI 192.168.203.47@tcp1 [ 1192.654609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.717526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1192.760675] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1193.321377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1193.340285] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1194.005485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1194.043409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.091439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1194.122923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1194.149699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1194.177441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1194.202335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1194.229529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.47@tcp [ 1194.246517] LNetError: 68834:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.47@tcp to recovery [ 1194.719584] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1197.622421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1200.863509] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1200.868474] LNet: Skipped 1 previous similar message [ 1205.275076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1205.309193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.345493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.775485] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:09:58 (1789186198) [ 1210.868081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.910387] LNet: 69545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.917663] LNetError: 69545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.940706] LNet: Removed LNI 192.168.203.47@tcp [ 1211.951460] LNet: Removed LNI 192.168.203.47@tcp1 [ 1211.992770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.047356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.078050] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1212.831355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.863367] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1213.592380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1213.663421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1213.685184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1213.723300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1213.760101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1213.800302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1213.849779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.47@tcp --health 0 [ 1213.934903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1214.047300] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.47@tcp [ 1214.049856] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1214.068804] LNet: Skipped 1 previous similar message [ 1228.910253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1229.035643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.062855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.094827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.47@tcp --health 500 [ 1229.170246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1235.690650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1235.757537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1235.782410] LNet: 70043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.795242] LNetError: 70043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.809335] LNetError: 70043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1235.819031] LNet: Removed LNI 192.168.203.47@tcp [ 1235.837504] LNet: Removed LNI 192.168.203.47@tcp1 [ 1235.863376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.912337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1235.929491] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1236.554308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.589067] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1237.269667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1237.302279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.369609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1237.413163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1237.457039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1237.479799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1237.509992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1237.547631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.47@tcp --health 0 [ 1238.943525] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1238.948123] LNet: Skipped 3 previous similar messages [ 1239.893606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1248.742870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1248.773236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.807027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.655808] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:10:41 (1789186241) [ 1253.959075] LNet: 71089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.964834] LNetError: 71089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.968697] LNetError: 71089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1255.007885] LNet: Removed LNI 192.168.203.47@tcp [ 1255.019857] LNet: Removed LNI 192.168.203.47@tcp1 [ 1255.464112] Key type .llcrypt unregistered [ 1255.467679] Key type ._llcrypt unregistered [ 1260.160392] Key type ._llcrypt registered [ 1260.162923] Key type .llcrypt registered [ 1260.334510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1260.353176] alg: No test for adler32 (adler32-zlib) [ 1261.248562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1261.272421] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1268.258919] Lustre: DEBUG MARKER: Initial discovery [ 1268.400518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.147@tcp [ 1270.159766] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1270.221960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1270.291707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.147@tcp [ 1270.327283] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1270.352238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.387388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.147@tcp -r 1 -m GET -l 3 [ 1270.418656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.147@tcp -r 1 -m GET -e local_error [ 1270.443596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.147@tcp -r 1 -m PUT -l 6 [ 1270.495113] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1271.414248] Lustre: DEBUG MARKER: Force oleg347-client.virtnet to discover 192.168.203.147@tcp (in background) [ 1271.571375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.147@tcp [ 1272.656494] Lustre: DEBUG MARKER: Force oleg347-server to discover 192.168.203.47@tcp [ 1274.659658] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1274.717717] Lustre: DEBUG MARKER: Wait for 71847 [ 1275.760709] Lustre: DEBUG MARKER: Finished wait on 71847 [ 1275.892962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1275.933347] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1275.945853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1276.193427] LNet: 72297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.202584] LNetError: 72297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.220484] LNet: Removed LNI 192.168.203.47@tcp [ 1276.504826] Key type .llcrypt unregistered [ 1276.506685] Key type ._llcrypt unregistered [ 1287.966768] Key type ._llcrypt registered [ 1287.968304] Key type .llcrypt registered [ 1288.861199] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:11:17 (1789186277) [ 1289.215563] Key type .llcrypt unregistered [ 1289.218576] Key type ._llcrypt unregistered [ 1293.327046] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1293.434372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1293.437982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1293.441924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1295.910414] Key type ._llcrypt registered [ 1295.920257] Key type .llcrypt registered [ 1296.137306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.151364] alg: No test for adler32 (adler32-zlib) [ 1297.055450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.108258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.135078] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1297.717808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1297.737267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1303.803634] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:11:32 (1789186292) [ 1303.922524] LNet: 74367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.926623] LNetError: 74367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1303.931245] LNet: Removed LNI 10.1.2.3@tcp [ 1303.937755] LNet: Removed LNI 192.168.203.47@tcp [ 1304.347394] Key type .llcrypt unregistered [ 1304.348819] Key type ._llcrypt unregistered [ 1308.473728] Key type ._llcrypt registered [ 1308.476239] Key type .llcrypt registered [ 1308.573950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1308.585403] alg: No test for adler32 (adler32-zlib) [ 1309.433885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1309.489438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1309.531815] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1309.924697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1309.938311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1310.533280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.47@tcp [ 1311.583225] LNetError: Unexpected error -22 connecting to 192.168.203.47@tcp at host 192.168.203.47:988 [ 1314.902317] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:11:43 (1789186303) [ 1315.035749] LNet: 75495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.043841] LNetError: 75495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1315.050612] LNet: Removed LNI 10.1.2.3@tcp [ 1316.072053] LNet: Removed LNI 192.168.203.47@tcp [ 1316.516044] Key type .llcrypt unregistered [ 1316.522149] Key type ._llcrypt unregistered [ 1320.607637] Key type ._llcrypt registered [ 1320.609706] Key type .llcrypt registered [ 1320.836971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.848600] alg: No test for adler32 (adler32-zlib) [ 1321.715645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.758786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.780422] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1322.279294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1322.297171] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1322.865595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.47@tcp1 --nid 192.168.203.47@tcp2 [ 1325.587910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp1 192.168.203.47@tcp2 [ 1325.609465] LNetError: 76327:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.47@tcp1 to 192.168.203.47@tcp2 - Route table may be misconfigured [ 1325.634831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp2 192.168.203.47@tcp1 [ 1329.552252] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:11:58 (1789186318) [ 1329.652249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1329.670245] LNet: 76899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.679700] LNetError: 76899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.719330] LNet: Removed LNI 192.168.203.47@tcp1 [ 1330.723452] LNetError: 76899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.749521] LNet: Removed LNI 192.168.203.47@tcp2 [ 1330.778806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.814379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1330.833096] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1331.299288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1331.313537] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1331.901981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1331.928924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.47@tcp [ 1331.957180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp1 192.168.203.47@tcp1 [ 1331.992907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.47@tcp -d 192.168.203.47@tcp -e network_timeout [ 1332.027590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -e network_timeout [ 1332.056726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -e network_timeout [ 1332.084822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -e network_timeout [ 1332.118452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1332.134803] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1332.162170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1332.256987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1332.320767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1332.831169] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1336.029990] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:12:04 (1789186324) [ 1336.177659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.204754] LNet: 77761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.213450] LNetError: 77761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.220064] LNet: Removed LNI 192.168.203.47@tcp [ 1337.254939] LNet: Removed LNI 192.168.203.47@tcp1 [ 1337.271804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.338567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1337.532660] LNet: 77799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.787521] Key type .llcrypt unregistered [ 1337.790097] Key type ._llcrypt unregistered [ 1344.517770] Key type ._llcrypt registered [ 1344.520763] Key type .llcrypt registered [ 1345.369736] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:12:13 (1789186333) [ 1345.550580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.561356] alg: No test for adler32 (adler32-zlib) [ 1346.424164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.337585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1348.463094] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:12:16 (1789186336) [ 1348.629684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1348.647996] LNet: 78894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.673286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.709967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.724931] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1349.287993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1349.307452] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1349.999516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1350.024723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp1 [ 1350.062433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp1 [ 1354.623227] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:12:23 (1789186343) [ 1355.840306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg347-server [ 1357.065537] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:12:25 (1789186345) [ 1358.242720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg347-server [ 1359.258949] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:12:27 (1789186347) [ 1360.390414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg347-server [ 1361.412253] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:12:29 (1789186349) [ 1362.638567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg347-server [ 1363.731086] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:12:32 (1789186352) [ 1365.203414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg347-server [ 1366.666705] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:12:34 (1789186354) [ 1368.105304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg347-server [ 1369.381433] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:12:37 (1789186357) [ 1370.781418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg347-server [ 1372.177297] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:12:40 (1789186360) [ 1373.462939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg347-server [ 1374.601173] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:12:43 (1789186363) [ 1375.907978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg347-server [ 1377.349476] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:12:45 (1789186365) [ 1377.524037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1377.542671] LNet: 82773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.545619] LNetError: 82773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.549644] LNet: Removed LNI 192.168.203.47@tcp [ 1378.599185] LNet: Removed LNI 192.168.203.47@tcp1 [ 1378.613263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.651085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.674926] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1379.191717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1379.237679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1379.382859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.396748] LNet: 82920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.404068] LNetError: 82920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.412060] LNetError: 82920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1379.428199] LNet: Removed LNI 192.168.203.47@tcp [ 1379.443633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.485652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1379.508930] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1380.009507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1380.144962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.221873] LNet: Removed LNI 192.168.203.47@tcp [ 1381.254801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.317844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.345648] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1381.892748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1381.937287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1383.190025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.205467] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.209351] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1383.215706] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.222239] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1383.239474] LNet: Removed LNI 192.168.203.47@tcp [ 1383.263467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.307033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1383.340812] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1383.892102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1384.053638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.135875] LNet: Removed LNI 192.168.203.47@tcp [ 1385.151633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.193229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.209426] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1385.693835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1385.764769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1385.920050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.937670] LNetError: 83514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.941164] LNetError: 83514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1385.950761] LNet: Removed LNI 192.168.203.47@tcp [ 1385.963892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.996329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1386.018701] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1386.620974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1386.793455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.877927] LNet: Removed LNI 192.168.203.47@tcp [ 1387.899634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.963933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.998604] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1388.576624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1388.641884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1388.799055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.811299] LNet: 83807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.815414] LNet: 83807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1389.870674] LNet: Removed LNI 192.168.203.47@tcp [ 1389.892801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.974977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1389.993242] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1390.623469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1390.833817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.855512] LNetError: 83952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.860172] LNetError: 83952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1390.869295] LNet: Removed LNI 192.168.203.47@tcp [ 1390.890626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.955292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1390.978562] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1391.434199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1391.481787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1391.612866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.677628] LNet: Removed LNI 192.168.203.47@tcp [ 1392.716492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.764593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1392.789625] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1393.347445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1393.491911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.534891] LNet: Removed LNI 192.168.203.47@tcp [ 1394.554602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.604354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1394.625104] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1395.074317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1395.137184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1395.328628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.392359] LNet: Removed LNI 192.168.203.47@tcp [ 1396.419025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.457847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1396.473808] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1397.046706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1397.261178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.275668] LNet: 84538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.280757] LNet: 84538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1398.311175] LNet: Removed LNI 192.168.203.47@tcp [ 1398.336750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.401162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1398.423286] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1398.981699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1399.011282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.029791] LNetError: 84670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.034357] LNetError: 84670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1399.046955] LNet: Removed LNI 192.168.203.47@tcp [ 1399.060512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.107095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.133218] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1399.631378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1399.698836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.730495] LNet: Removed LNI 192.168.203.47@tcp [ 1399.743090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.781446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1399.856986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.883876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.924396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1399.979860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.003495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.053830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1400.116604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.143254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.186232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1400.204707] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1400.729389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.754685] LNet: Removed LNI 192.168.203.47@tcp [ 1400.784488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.851653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1405.558626] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:13:13 (1789186393) [ 1405.630685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1405.656871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.705805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1405.723645] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/180] [ 1406.507351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.530429] LNet: Removed LNI 192.168.203.47@tcp231 [ 1406.547721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.596882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1406.616708] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/180] [ 1406.623635] LNet: 85704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1406.915533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1406.945048] LNet: Removed LNI 192.168.203.47@tcp231 [ 1406.961022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1406.978424] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/180] [ 1407.680342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.705235] LNet: Removed LNI 192.168.203.47@tcp231 [ 1407.718881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.774363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1407.799549] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/0] [ 1407.807342] LNet: 85857:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.044306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1408.063311] LNet: Removed LNI 192.168.203.47@tcp231 [ 1408.071150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1408.083276] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/0] [ 1408.813216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.848443] LNet: Removed LNI 192.168.203.47@tcp231 [ 1408.867476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.917045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1408.938333] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/60] [ 1408.944491] LNet: 86008:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1409.260818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1409.285722] LNet: Removed LNI 192.168.203.47@tcp231 [ 1409.294374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1409.317268] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/60] [ 1409.982446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.999625] LNet: Removed LNI 192.168.203.47@tcp231 [ 1410.014960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.058720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1410.077966] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/180] [ 1410.359769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1410.392372] LNet: Removed LNI 192.168.203.47@tcp231 [ 1410.406911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1410.436069] LNet: Added LNI 192.168.203.47@tcp231 [8/256/0/180] [ 1415.189757] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:13:23 (1789186403) [ 1415.241968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.258686] LNet: 86863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.265283] LNet: 86863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1415.276638] LNetError: 86863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.283712] LNetError: 86863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1415.307475] LNet: Removed LNI 192.168.203.47@tcp231 [ 1415.324148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.362455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1415.377725] LNet: Added LNI 192.168.203.47@tcp232 [8/256/0/180] [ 1415.816407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1415.845826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1415.875207] LNet: Removed LNI 192.168.203.47@tcp232 [ 1415.893522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1415.913892] LNet: Added LNI 192.168.203.47@tcp232 [8/256/0/180] [ 1416.368646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1416.388941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1416.417372] LNet: Removed LNI 192.168.203.47@tcp232 [ 1416.435104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1416.456339] LNet: Added LNI 192.168.203.47@tcp232 [8/256/0/180] [ 1416.847374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1416.875493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1416.901271] LNet: Removed LNI 192.168.203.47@tcp232 [ 1416.913196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1416.929881] LNet: Added LNI 192.168.203.47@tcp232 [8/256/0/180] [ 1417.317488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1417.344477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1417.368920] LNet: Removed LNI 192.168.203.47@tcp232 [ 1417.380509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1420.797440] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:13:29 (1789186409) [ 1421.410309] Key type .llcrypt unregistered [ 1421.413528] Key type ._llcrypt unregistered [ 1424.930495] Key type ._llcrypt registered [ 1424.935318] Key type .llcrypt registered [ 1425.060279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1425.078310] alg: No test for adler32 (adler32-zlib) [ 1425.934393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1425.968465] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1431.482925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1436.057791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.089651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.108518] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1436.784330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_interrupt -r 2 [ 1436.811952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_interrupt -r 2 [ 1436.831670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_interrupt -r 2 [ 1436.851874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_interrupt -r 2 [ 1436.872989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_interrupt -r 2 [ 1436.891652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_interrupt -r 2 [ 1436.917033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_interrupt -r 2 [ 1436.942880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_interrupt -r 2 [ 1436.964713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1436.981775] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1436.985817] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1437.110384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1437.124574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1437.145479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1437.163859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1437.183306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1437.200222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1437.219420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1437.241845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1437.261254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1437.271815] LNet: 88656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1437.284174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1437.299373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1437.317520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.335979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1437.352676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1437.371341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.634250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_dropped -r 2 [ 1437.654477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_dropped -r 2 [ 1437.677318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_dropped -r 2 [ 1437.701650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_dropped -r 2 [ 1437.719779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_dropped -r 2 [ 1437.738290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_dropped -r 2 [ 1437.757391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_dropped -r 2 [ 1437.777925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_dropped -r 2 [ 1437.800094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1437.811262] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1437.946828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1437.968902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1437.992208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1438.011567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1438.031797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1438.049422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1438.066456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1438.080047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1438.095674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1438.105116] LNet: 88778:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1438.116871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1438.130472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1438.145653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1438.161613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.178738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1438.196087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1438.430628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_aborted -r 2 [ 1438.453443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_aborted -r 2 [ 1438.473854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_aborted -r 2 [ 1438.493379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_aborted -r 2 [ 1438.514965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_aborted -r 2 [ 1438.540783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_aborted -r 2 [ 1438.563809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_aborted -r 2 [ 1438.585717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_aborted -r 2 [ 1438.606774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1438.724582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1438.746321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1438.763140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1438.780730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1438.796147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1438.813737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1438.830738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1438.848574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1438.866536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1438.883729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1438.905771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1438.925560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1438.949116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.972352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1438.989619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1439.241574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_no_route -r 2 [ 1439.261978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_no_route -r 2 [ 1439.282367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_no_route -r 2 [ 1439.305505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_no_route -r 2 [ 1439.331502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_no_route -r 2 [ 1439.351037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_no_route -r 2 [ 1439.371375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_no_route -r 2 [ 1439.393689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_no_route -r 2 [ 1439.425073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1439.445102] LNet: There was an unexpected network error while writing to 192.168.203.147: rc = -22 [ 1439.449515] LNet: Skipped 2 previous similar messages [ 1439.593815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1439.614292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1439.633524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1439.654919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1439.670626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1439.689923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1439.710252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1439.729334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1439.746664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1439.758122] LNet: 89023:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1439.761284] LNet: 89023:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1439.772453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1439.789645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1439.807214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1439.831410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1439.854326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1439.874609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1440.149077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e local_timeout -r 2 [ 1440.177192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e local_timeout -r 2 [ 1440.200505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e local_timeout -r 2 [ 1440.223445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e local_timeout -r 2 [ 1440.245500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e local_timeout -r 2 [ 1440.267302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e local_timeout -r 2 [ 1440.288733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e local_timeout -r 2 [ 1440.310983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e local_timeout -r 2 [ 1440.336699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1440.469887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1440.493836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1440.519116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1440.541050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1440.561066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1440.578692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1440.603220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1440.622879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.642084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.660795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.679925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.705808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.725099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.745657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.766349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1441.030691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp -m GET -e remote_dropped -r 2 [ 1441.055321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp -d 192.168.203.147@tcp1 -m GET -e remote_dropped -r 2 [ 1441.084090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp -m GET -e remote_dropped -r 2 [ 1441.109797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp -d 192.168.203.47@tcp1 -m GET -e remote_dropped -r 2 [ 1441.134240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp -m GET -e remote_dropped -r 2 [ 1441.159576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.147@tcp1 -m GET -e remote_dropped -r 2 [ 1441.187884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp -m GET -e remote_dropped -r 2 [ 1441.215164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.47@tcp1 -d 192.168.203.47@tcp1 -m GET -e remote_dropped -r 2 [ 1441.242070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.47@tcp 192.168.203.147@tcp [ 1441.261195] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.147@tcp [ 1441.379500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1441.398454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1441.417127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1441.434583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1441.457786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1441.473889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1441.497759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1441.513816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1441.537389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1441.561816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1441.584053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1441.603878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1441.624257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1441.639572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1441.657589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.549218] LNet: 89341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.555402] LNetError: 89341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.560300] LNet: Removed LNI 192.168.203.47@tcp [ 1444.572120] LNet: Removed LNI 192.168.203.47@tcp1 [ 1444.857643] Key type .llcrypt unregistered [ 1444.860166] Key type ._llcrypt unregistered [ 1454.299817] Key type ._llcrypt registered [ 1454.303077] Key type .llcrypt registered [ 1455.111616] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:14:03 (1789186443) [ 1455.410930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1455.423460] alg: No test for adler32 (adler32-zlib) [ 1456.250634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.283212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.298418] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1456.778281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1456.802559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.47@tcp [ 1456.817083] LNetError: 90316:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.47@tcp to recovery [ 1456.870432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.47@tcp [ 1456.927697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1460.393436] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:14:08 (1789186448) [ 1460.512117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.524486] LNet: 90905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.528375] LNetError: 90905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.544033] LNet: Removed LNI 192.168.203.47@tcp [ 1460.560432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.599078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.616905] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1461.089456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1461.104613] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1461.755961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.47@tcp [ 1461.777152] LNetError: 91171:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.47@tcp to recovery [ 1462.623379] LNet: 1 local NIs in recovery (showing 1): 192.168.203.47@tcp [ 1462.624358] LNet: There was an unexpected network error while writing to 192.168.203.47: rc = -22 [ 1462.924308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.47@tcp [ 1469.171381] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:14:17 (1789186457) [ 1470.292091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg347-server [ 1471.397335] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:14:19 (1789186459) [ 1472.682761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg347-server [ 1473.751504] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:14:22 (1789186462) [ 1474.736671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg347-server [ 1475.681496] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:14:24 (1789186464) [ 1475.919277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.938462] LNet: 92801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.945105] LNetError: 92801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.950830] LNet: Removed LNI 192.168.203.47@tcp [ 1475.965580] LNet: Removed LNI 192.168.203.47@tcp1 [ 1475.983128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.014727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.030144] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1476.458653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1476.495628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1476.520347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1476.590373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1476.615205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1476.672269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1479.911706] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:14:28 (1789186468) [ 1480.079952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.096144] LNet: 93531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.100310] LNetError: 93531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.104518] LNetError: 93531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1480.119578] LNet: Removed LNI 192.168.203.47@tcp [ 1480.130575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.172908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.189747] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1484.647278] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:14:33 (1789186473) [ 1484.771342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.795150] LNet: 94234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.807099] LNetError: 94234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.822605] LNet: Removed LNI 192.168.203.47@tcp [ 1484.837210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.893603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.908141] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1489.538282] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:14:37 (1789186477) [ 1490.511405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1491.740460] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:14:40 (1789186480) [ 1492.241436] LNet: 95269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.247845] LNetError: 95269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.264581] LNet: Removed LNI 192.168.203.47@tcp [ 1492.791722] Key type .llcrypt unregistered [ 1492.794113] Key type ._llcrypt unregistered [ 1496.903523] Key type ._llcrypt registered [ 1496.906880] Key type .llcrypt registered [ 1497.049384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1497.059730] alg: No test for adler32 (adler32-zlib) [ 1497.912916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1497.945368] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1503.133808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1506.156380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.147@tcp [ 1506.404881] LNet: 95682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.411576] LNetError: 95682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.429639] LNet: Removed LNI 192.168.203.47@tcp [ 1506.731386] Key type .llcrypt unregistered [ 1506.732956] Key type ._llcrypt unregistered [ 1513.764420] Key type ._llcrypt registered [ 1513.766214] Key type .llcrypt registered [ 1514.570655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1515.521810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1516.610559] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:15:05 (1789186505) [ 1517.516916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1518.591189] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:15:07 (1789186507) [ 1519.757208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg347-server [ 1520.839169] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:15:09 (1789186509) [ 1521.860338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg347-server [ 1522.857982] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:15:11 (1789186511) [ 1523.741762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.754990] alg: No test for adler32 (adler32-zlib) [ 1524.674372] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1524.813686] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1526.431261] Key type lgssc registered [ 1527.168870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.614849] Key type lgssc unregistered [ 1539.790891] LNet: 98850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.794990] LNetError: 98850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.810894] LNet: Removed LNI 192.168.203.47@tcp [ 1540.263526] Key type .llcrypt unregistered [ 1540.265890] Key type ._llcrypt unregistered [ 1548.614335] Key type ._llcrypt registered [ 1548.616303] Key type .llcrypt registered [ 1549.411942] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:15:37 (1789186537) [ 1549.662286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1549.678053] alg: No test for adler32 (adler32-zlib) [ 1550.535544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.601535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1550.639864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1550.671880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1550.702798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.351584] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:15:42 (1789186542) [ 1554.408470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.426769] LNet: 100255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.445371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.506795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.532822] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1555.139585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.153140] LNet: 100405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.162396] LNetError: 100405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.176528] LNet: Removed LNI 192.168.203.47@tcp [ 1555.191802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.235772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1555.265207] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1555.552397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.583488] LNet: Removed LNI 192.168.203.47@tcp [ 1555.599110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.650590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1555.671580] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1559.301271] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:15:47 (1789186547) [ 1559.380030] LNet: 101022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.383874] LNet: 101022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1559.389669] LNetError: 101022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.394091] LNetError: 101022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1559.409282] LNet: Removed LNI 192.168.203.47@tcp [ 1559.749822] Key type .llcrypt unregistered [ 1559.755240] Key type ._llcrypt unregistered [ 1563.075725] Key type ._llcrypt registered [ 1563.079895] Key type .llcrypt registered [ 1563.198828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.211625] alg: No test for adler32 (adler32-zlib) [ 1564.080237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.128559] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1564.131273] LNet: *** cfs_fail_loc=e003, val=100*** [ 1564.556196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1564.573049] Ping reply received with mapping: [ 1564.575045] found metadata reply NI[192.168.203.47@tcp] [ 1564.577310] 2f cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1564.580536] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1564.583385] 5a 5a 5a 5a 5a 5a 5a 5a [ 1564.668784] LNet: 101417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.674976] LNetError: 101417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.685901] LNet: Removed LNI 192.168.203.47@tcp [ 1565.016457] Key type .llcrypt unregistered [ 1565.019701] Key type ._llcrypt unregistered [ 1571.233598] Key type ._llcrypt registered [ 1571.234993] Key type .llcrypt registered [ 1572.061550] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:16:00 (1789186560) [ 1572.376594] Key type .llcrypt unregistered [ 1572.378080] Key type ._llcrypt unregistered [ 1576.129798] Key type ._llcrypt registered [ 1576.132785] Key type .llcrypt registered [ 1576.262577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1576.277888] alg: No test for adler32 (adler32-zlib) [ 1577.248374] LNetError: 102466:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1577.563663] Key type .llcrypt unregistered [ 1577.567184] Key type ._llcrypt unregistered [ 1581.192513] Key type ._llcrypt registered [ 1581.198250] Key type .llcrypt registered [ 1582.083159] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:16:10 (1789186570) [ 1582.426453] Key type .llcrypt unregistered [ 1582.430837] Key type ._llcrypt unregistered [ 1586.174405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1586.177805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1588.452496] Key type ._llcrypt registered [ 1588.459743] Key type .llcrypt registered [ 1588.581290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1588.589932] alg: No test for adler32 (adler32-zlib) [ 1589.441490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.484844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1595.992892] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:16:24 (1789186584) [ 1596.091665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.119628] LNet: 104236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.127887] LNetError: 104236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1596.141260] LNet: Removed LNI 10.1.2.3@tcp [ 1596.160114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.666891] Key type .llcrypt unregistered [ 1596.669568] Key type ._llcrypt unregistered [ 1600.628646] Key type ._llcrypt registered [ 1600.630636] Key type .llcrypt registered [ 1600.761207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.778839] alg: No test for adler32 (adler32-zlib) [ 1601.659843] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1601.715736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1601.833227] LNet: 104523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.838698] LNetError: 104523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.853609] LNet: Removed LNI 192.168.203.47@tcp [ 1603.297484] Key type .llcrypt unregistered [ 1603.302083] Key type ._llcrypt unregistered [ 1610.353885] Key type ._llcrypt registered [ 1610.359705] Key type .llcrypt registered [ 1611.182595] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:16:39 (1789186599) [ 1611.531287] Key type .llcrypt unregistered [ 1611.533134] Key type ._llcrypt unregistered [ 1615.505576] Key type ._llcrypt registered [ 1615.507059] Key type .llcrypt registered [ 1615.632821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.641588] alg: No test for adler32 (adler32-zlib) [ 1620.621916] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:16:48 (1789186608) [ 1620.691737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.731231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.784501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.803949] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1621.279922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.304462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1621.326971] LNet: Removed LNI 192.168.203.47@tcp [ 1621.486611] LNet: 106261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.764578] Key type .llcrypt unregistered [ 1621.767788] Key type ._llcrypt unregistered [ 1629.538774] Key type ._llcrypt registered [ 1629.540657] Key type .llcrypt registered [ 1630.317754] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:16:58 (1789186618) [ 1630.576786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.598472] alg: No test for adler32 (adler32-zlib) [ 1631.463852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.523787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.541575] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1632.035195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1632.056318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.47@tcp [ 1632.076482] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1632.080575] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1632.086535] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1632, 0, 0 [ 1632.092476] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1632.095140] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1632.099082] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1632.101977] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1632, 0, 0 [ 1632.107610] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1632.110972] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1632.114667] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1632.117105] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1632, 0, 1682 [ 1632.121394] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1632.123150] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1632.125627] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1632.129091] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1632, 0, 1682 [ 1632.133265] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1635.618068] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:17:04 (1789186624) [ 1635.882898] LNet: 107797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.889270] LNetError: 107797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.905391] LNet: Removed LNI 192.168.203.47@tcp [ 1636.293663] Key type .llcrypt unregistered [ 1636.295609] Key type ._llcrypt unregistered [ 1639.611305] Key type ._llcrypt registered [ 1639.616862] Key type .llcrypt registered [ 1639.728208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.737464] alg: No test for adler32 (adler32-zlib) [ 1640.569939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1640.589477] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1644.941545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1650.828928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.862594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.886957] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1651.545538] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1651.656245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1651.660184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1651.663730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1653.915050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1653.935051] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1660.995573] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:17:29 (1789186649) [ 1661.174645] LNet: 109296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.181131] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.186957] LNet: Removed LNI 192.168.203.47@tcp [ 1662.239181] LNet: Removed LNI 192.168.203.47@tcp1 [ 1662.242325] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1662.248342] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1662.263967] LNet: Removed LNI 10.1.2.3@tcp99 [ 1662.687811] Key type .llcrypt unregistered [ 1662.690717] Key type ._llcrypt unregistered [ 1666.347962] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1666.352080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1668.665700] Key type ._llcrypt registered [ 1668.667307] Key type .llcrypt registered [ 1668.792431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1668.803454] alg: No test for adler32 (adler32-zlib) [ 1669.659796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.704387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.723549] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1670.161863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1670.183595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1671.466897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1671.496207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.524301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1671.542418] LNetError: 110038:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1671.565280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.612901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1671.851815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.910209] LNet: 110058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.916453] LNetError: 110058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1671.920521] LNet: Removed LNI 10.1.2.3@tcp [ 1672.935520] LNet: Removed LNI 192.168.203.47@tcp [ 1673.318427] Key type .llcrypt unregistered [ 1673.320468] Key type ._llcrypt unregistered [ 1679.742961] Key type ._llcrypt registered [ 1679.745477] Key type .llcrypt registered [ 1680.573637] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:17:48 (1789186668) [ 1680.890222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.903609] alg: No test for adler32 (adler32-zlib) [ 1681.771278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.800174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.824323] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1682.314377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1682.367531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg347-client@tcp [ 1686.579723] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:17:54 (1789186674) [ 1686.653630] LNet: 111581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.660724] LNetError: 111581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.681750] LNet: Removed LNI 192.168.203.47@tcp [ 1687.081359] Key type .llcrypt unregistered [ 1687.084679] Key type ._llcrypt unregistered [ 1690.580753] Key type ._llcrypt registered [ 1690.582468] Key type .llcrypt registered [ 1690.673522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1690.687586] alg: No test for adler32 (adler32-zlib) [ 1691.522409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1691.541916] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1691.559871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1695.360908] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:18:03 (1789186683) [ 1696.200526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1697.267140] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:18:05 (1789186685) [ 1698.133877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1699.337688] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:18:07 (1789186687) [ 1699.421734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.443403] LNet: 113005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.448837] LNetError: 113005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.462605] LNet: Removed LNI 192.168.203.47@tcp [ 1699.476978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.520053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.537252] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1700.026820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1700.054793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1704.436587] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:18:12 (1789186692) [ 1704.536396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.561039] LNet: 113714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.568398] LNetError: 113714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.586985] LNet: Removed LNI 192.168.203.47@tcp [ 1704.604238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.649453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1704.682617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1705.205569] Key type .llcrypt unregistered [ 1705.208560] Key type ._llcrypt unregistered [ 1713.166596] Key type ._llcrypt registered [ 1713.169957] Key type .llcrypt registered [ 1714.130739] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:18:22 (1789186702) [ 1714.364509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.374828] alg: No test for adler32 (adler32-zlib) [ 1715.266996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.315885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.331574] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1715.791839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1715.832451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1719.915761] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:18:28 (1789186708) [ 1720.019911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.040686] LNet: 115266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.050434] LNetError: 115266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.066074] LNet: Removed LNI 192.168.203.47@tcp [ 1720.077601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.105607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1720.128380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1724.427868] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:18:32 (1789186712) [ 1724.592274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.628755] LNet: 115846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.646678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.690844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1724.916506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1724.944391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1724.989546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1725.283502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1725.309723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1725.370641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.397189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.449929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1725.653219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1725.676089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1725.724059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1726.018798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1726.046256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1730.334383] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:18:38 (1789186718) [ 1730.492700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.509490] LNet: 116479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.515098] LNet: 116479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1730.537305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.585088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.606919] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1731.051675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1731.068880] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1731.603817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1731.652486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1731.810672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.838555] LNetError: 116759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.845444] LNet: Removed LNI 192.168.203.47@tcp [ 1731.859242] LNet: Removed LNI 192.168.203.47@tcp2 [ 1731.878569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.920593] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1731.925271] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1731.928826] LNet: 116765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1732.224375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1732.366795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.407142] LNet: Removed LNI 192.168.203.47@tcp [ 1733.410233] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.416911] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1733.430788] LNet: Removed LNI 192.168.203.47@tcp2 [ 1733.459881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.511241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.534499] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1734.104964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1734.131245] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1734.743407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1734.783468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1734.845798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1734.999760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.013102] LNet: 117086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.016934] LNet: 117086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1736.031664] LNet: Removed LNI 192.168.203.47@tcp [ 1736.034054] LNetError: 117086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.043343] LNetError: 117086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1736.060598] LNet: Removed LNI 192.168.203.47@tcp2 [ 1736.083377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.136929] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1736.141783] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1736.145417] LNet: 117092:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1736.483287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1736.671948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.689426] LNet: Removed LNI 192.168.203.47@tcp [ 1737.702466] LNet: Removed LNI 192.168.203.47@tcp2 [ 1737.728732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.775597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.798318] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1738.335955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1738.348817] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1738.981340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1739.032845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1739.188885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.208370] LNet: 117409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.213522] LNet: 117409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1740.260771] LNet: Removed LNI 192.168.203.47@tcp [ 1740.265642] LNetError: 117409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.273914] LNetError: 117409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1740.293189] LNet: Removed LNI 192.168.203.47@tcp2 [ 1740.313103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.393536] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1740.403767] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1740.417904] LNet: 117415:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.845045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1741.004220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.047222] LNet: Removed LNI 192.168.203.47@tcp [ 1742.056929] LNet: Removed LNI 192.168.203.47@tcp2 [ 1742.078724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.140553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.154119] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1742.667516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1742.690495] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1743.337501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1743.364708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1743.403490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1743.530210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.607237] LNet: Removed LNI 192.168.203.47@tcp [ 1745.640895] LNet: Removed LNI 192.168.203.47@tcp2 [ 1745.660081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.705836] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1745.712452] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1745.716254] LNet: 117743:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.038450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1746.187483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.208672] LNet: Removed LNI 192.168.203.47@tcp [ 1747.242843] LNet: Removed LNI 192.168.203.47@tcp2 [ 1747.263832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.315874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.333840] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1747.819714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1747.840905] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1748.418590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1748.443676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1748.486631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1748.701363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.722572] LNet: 118077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.728574] LNet: 118077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1748.733496] LNetError: 118077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.737321] LNetError: 118077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1749.792386] LNet: Removed LNI 192.168.203.47@tcp [ 1749.809191] LNet: Removed LNI 192.168.203.47@tcp2 [ 1749.840379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.913030] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1749.921635] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1749.931508] LNet: 118084:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.338345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1750.575610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.647158] LNet: Removed LNI 192.168.203.47@tcp [ 1751.658535] LNet: Removed LNI 192.168.203.47@tcp2 [ 1751.679608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.716572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.741312] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1752.194496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1752.208962] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1752.783943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1752.812288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1752.836577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1752.875318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1753.081502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.143145] LNet: Removed LNI 192.168.203.47@tcp [ 1755.178586] LNet: Removed LNI 192.168.203.47@tcp2 [ 1755.197635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.249829] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1755.252882] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1755.546081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1759.589866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1760.629083] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:19:09 (1789186749) [ 1760.705055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.731447] LNet: Removed LNI 192.168.203.47@tcp [ 1761.767620] LNet: Removed LNI 192.168.203.47@tcp2 [ 1761.787619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.951857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.969896] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1762.442283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1762.457913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1763.020797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.040848] LNet: Removed LNI 10.1.2.3@tcp [ 1763.053261] LNet: Removed LNI 192.168.203.47@tcp [ 1763.066672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.114384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1763.137481] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1763.141990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1763.416816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1763.446956] LNet: Removed LNI 192.168.203.47@tcp [ 1763.459182] LNet: Removed LNI 10.1.2.3@tcp [ 1767.622322] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:19:16 (1789186756) [ 1767.765345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.781855] LNet: 120076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.788368] LNet: 120076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1767.808158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.862191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.883899] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1768.377727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1768.409651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1768.444987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.473568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.508581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.538269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.580456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.610710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.641960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.666718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.687938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.713629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1768.850880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1773.032633] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 00:19:21 (1789186761) [ 1773.102277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.124443] LNetError: 120809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.129547] LNetError: 120809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1773.143128] LNet: Removed LNI 192.168.203.47@tcp [ 1773.160730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.208512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.233019] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1773.715529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 1773.736417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1773.767075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.789512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.824042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.859658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.885368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.910955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.939490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.968769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1773.991810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.011994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.073471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1774.101000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.124808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.151668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.175959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.201189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.226742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.264632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.303762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.328619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1774.350632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.47@tcp [ 1778.301708] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:19:26 (1789186766) [ 1778.391898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.432066] LNet: Removed LNI 192.168.203.47@tcp [ 1779.453914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.531360] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1779.669997] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1779.673068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1779.675936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1781.846494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.870548] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1782.273611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1782.287233] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1783.515555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1783.544715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1783.586835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.47@tcp -m PUT -e local_timeout -r 1 [ 1783.633931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1783.674091] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1783.677383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1783.832587] LNetError: 121635:0:(lib-move.c:4794:lnet_parse()) 192.168.203.47@tcp, src 192.168.203.47@tcp: Dropping PUT (error -108 looking up sender) [ 1784.863773] LNet: Removed LNI 10.1.2.3@tcp [ 1785.892520] LNet: Removed LNI 192.168.203.47@tcp [ 1786.120784] Key type .llcrypt unregistered [ 1786.122745] Key type ._llcrypt unregistered [ 1792.839361] Key type ._llcrypt registered [ 1792.841776] Key type .llcrypt registered [ 1793.568582] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:19:42 (1789186782) [ 1793.806604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1793.815363] alg: No test for adler32 (adler32-zlib) [ 1794.662638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.870846] LNet: 122944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.225857] Key type .llcrypt unregistered [ 1795.228378] Key type ._llcrypt unregistered [ 1798.874404] Key type ._llcrypt registered [ 1798.878712] Key type .llcrypt registered [ 1798.996679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1799.008393] alg: No test for adler32 (adler32-zlib) [ 1799.844380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1799.868862] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1804.216186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1804.346643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1804.689259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.147@tcp [ 1804.834832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1804.856631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1804.878214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1804.896242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1804.914887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1804.933497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1804.955685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1804.981657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.005942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.021522] LNet: 123392:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.024223] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.147@tcp [ 1805.041041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.064203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.099629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.127866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.154473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.176424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1805.547089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1805.680738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1805.703467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.726186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.750825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.768534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1805.786796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.812299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.844026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.866086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.878303] LNet: 123511:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.891234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.922600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.944547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.967357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.985940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.007651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1806.400606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.147@tcp [ 1806.537524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1806.553949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1806.575925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1806.600871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1806.620691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1806.642336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1806.661647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1806.681669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1806.702417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1806.719628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1806.743667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1806.767850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.792339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1806.822249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.840928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.202267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1807.339925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.363923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.384873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.409447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.435757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1807.460933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.493642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.515310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.532140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.543749] LNet: 123750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.547966] LNet: 123750:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1807.560586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1807.581813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1807.601219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1807.620293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.638824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1807.665232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.836667] LNet: 123796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.841297] LNetError: 123796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.857532] LNet: Removed LNI 192.168.203.47@tcp [ 1808.043819] Key type .llcrypt unregistered [ 1808.045374] Key type ._llcrypt unregistered [ 1818.020131] Key type ._llcrypt registered [ 1818.022261] Key type .llcrypt registered [ 1818.841524] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:20:07 (1789186807) [ 1819.132705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.146605] alg: No test for adler32 (adler32-zlib) [ 1820.034125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.301974] LNet: 124682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.682171] Key type .llcrypt unregistered [ 1820.684610] Key type ._llcrypt unregistered [ 1824.853790] Key type ._llcrypt registered [ 1824.856460] Key type .llcrypt registered [ 1824.992810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.002956] alg: No test for adler32 (adler32-zlib) [ 1825.871435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1825.891516] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1831.422536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.147@tcp [ 1837.071612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.107049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1837.126349] LNet: Added LNI 192.168.203.47@tcp1 [8/256/0/180] [ 1837.829614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1838.182025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.147@tcp [ 1838.327539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.346641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1838.369270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.397743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1838.422496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1838.448294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1838.474817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1838.506788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1838.532836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1838.545880] LNet: 125333:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.566244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1838.588959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1838.607871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.628534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.644849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.663606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1839.013882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1839.163569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.182124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1839.199180] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.147@tcp, 192.168.203.147@tcp1 [ 1839.206420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.223714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.247069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1839.272326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.300175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.331919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.362418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.377479] LNet: 125452:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1839.394763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.422738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.445981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.469456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1839.490729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1839.515119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1839.878853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.147@tcp [ 1840.022739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.038418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1840.058361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1840.074781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1840.101927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1840.126127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1840.152264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1840.173635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.200491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.227707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.253210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.275765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.299607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.328811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.352138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.749776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1840.930553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.959105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1840.990140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.018273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1841.047131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1841.080491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1841.104160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1841.128205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1841.150407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1841.164794] LNet: 125690:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1841.170174] LNet: 125690:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1841.183443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1841.206981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1841.228615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1841.254996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1841.282538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1841.308952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1844.648429] LNet: 125771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.655695] LNetError: 125771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.663384] LNet: Removed LNI 192.168.203.47@tcp [ 1844.671690] LNet: Removed LNI 192.168.203.47@tcp1 [ 1844.931408] Key type .llcrypt unregistered [ 1844.934298] Key type ._llcrypt unregistered [ 1854.950614] Key type ._llcrypt registered [ 1854.953290] Key type .llcrypt registered [ 1855.827339] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:20:44 (1789186844) [ 1856.061654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.072418] alg: No test for adler32 (adler32-zlib) [ 1856.967642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.010650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1857.032071] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1857.516784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1857.541111] LNet: Added LNI 192.168.203.47@tcp2 [8/256/0/180] [ 1858.134549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.47@tcp [ 1858.191726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1859.231332] LNet: Removed LNI 192.168.203.47@tcp [ 1863.649046] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:20:52 (1789186852) [ 1864.664342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1865.797283] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:20:54 (1789186854) [ 1870.472939] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:20:58 (1789186858) [ 1875.007947] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:21:03 (1789186863) [ 1879.163995] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:21:07 (1789186867) [ 1883.295482] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:21:11 (1789186871) [ 1887.740321] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:21:16 (1789186876) [ 1892.422131] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:21:20 (1789186880) [ 1897.325861] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:21:25 (1789186885) [ 1901.424486] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:21:29 (1789186889) [ 1905.791796] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:21:34 (1789186894) [ 1909.910516] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:21:38 (1789186898) [ 1913.799258] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:21:42 (1789186902) [ 1917.443883] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:21:46 (1789186906) [ 1921.375972] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:21:49 (1789186909) [ 1925.769831] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:21:54 (1789186914) [ 1929.590522] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:21:58 (1789186918) [ 1933.781714] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:22:02 (1789186922) [ 1937.756576] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:22:06 (1789186926) [ 1937.920894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.946540] LNet: 137453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.952645] LNetError: 137453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.962099] LNet: Removed LNI 192.168.203.47@tcp2 [ 1937.977515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.046634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.064987] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1938.741454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.755380] LNet: 137595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.760634] LNetError: 137595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.773875] LNet: Removed LNI 192.168.203.47@tcp [ 1938.791236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.830545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1938.854313] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1942.467380] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:22:11 (1789186931) [ 1946.672230] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:22:15 (1789186935) [ 1950.784823] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:22:19 (1789186939) [ 1954.732457] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:22:23 (1789186943) [ 1958.102674] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:22:26 (1789186946) [ 1961.993734] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:22:30 (1789186950) [ 1965.861325] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:22:34 (1789186954) [ 1969.648316] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:22:38 (1789186958) [ 1973.335545] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:22:41 (1789186961) [ 1977.639354] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:22:46 (1789186966) [ 1981.758516] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:22:50 (1789186970) [ 1985.771024] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:22:54 (1789186974) [ 1989.531662] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1800 sec ========== 00:22:58 (1789186978) [ 1989.708179] LNet: 145053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.712159] LNetError: 145053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.722142] LNet: Removed LNI 192.168.203.47@tcp [ 1990.061680] Key type .llcrypt unregistered [ 1990.063521] Key type ._llcrypt unregistered [ 1996.257063] Key type ._llcrypt registered [ 1996.258666] Key type .llcrypt registered [ 1996.492435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.499543] alg: No test for adler32 (adler32-zlib) [ 1997.431393] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1997.585313] LNet: Added LNI 192.168.203.47@tcp [8/256/0/180] [ 1999.207696] Key type lgssc registered [ 1999.861485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2038.184867] Lustre: Mounted lustre-client - version 2.17.58_40_g357d090 [ 2041.354322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2045.740639] Lustre: Unmounted lustre-client [ 2077.413144] Key type lgssc unregistered [ 2077.599339] LNet: 147996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.603672] LNetError: 147996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.621668] LNet: Removed LNI 192.168.203.47@tcp [ 2077.988173] Key type .llcrypt unregistered [ 2077.990299] Key type ._llcrypt unregistered