[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455165820 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003331] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007960] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022469] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027249] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031140] *** VALIDATE tmpfs *** [ 0.033298] *** VALIDATE proc *** [ 0.034307] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037120] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044425] debug: unmapping init [mem 0xffffffff8cc59000-0xffffffff8cc60fff] [ 0.046196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052020] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054016] ... event mask: 000000070000000f [ 0.055338] rcu: Hierarchical SRCU implementation. [ 0.057494] smp: Bringing up secondary CPUs ... [ 0.058607] x86: Booting SMP configuration: [ 0.059034] .... node #0, CPUs: #1 #2 #3 [ 0.062555] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131272] node 0 deferred pages initialised in 64ms [ 0.134375] devtmpfs: initialized [ 0.136283] x86/mm: Memory block size: 128MB [ 0.139807] gcov: version magic: 0x41383552 [ 0.142287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149348] pinctrl core: initialized pinctrl subsystem [ 0.152211] [ 0.152891] ************************************************************* [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.160014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163017] ** ** [ 0.166014] ** This means that this kernel is built to expose internal ** [ 0.168024] ** IOMMU data structures, which may compromise security on ** [ 0.171012] ** your system. ** [ 0.174015] ** ** [ 0.176009] ** If you see this message and you are not debugging the ** [ 0.178012] ** kernel, report this immediately to your vendor! ** [ 0.180014] ** ** [ 0.182011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185013] ************************************************************* [ 0.188712] NET: Registered protocol family 16 [ 0.190452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201031] cpuidle: using governor menu [ 0.202740] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205606] PCI: Using configuration type 1 for base access [ 0.208221] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220134] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227050] cryptd: max_cpu_qlen set to 1000 [ 0.229312] ACPI: Added _OSI(Module Device) [ 0.230025] ACPI: Added _OSI(Processor Device) [ 0.231023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232042] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237161] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245443] ACPI: Interpreter enabled [ 0.247112] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248013] ACPI: Using IOAPIC for interrupt routing [ 0.250140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262886] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276427] acpiphp: Slot [2] registered [ 0.278197] acpiphp: Slot [5] registered [ 0.279218] acpiphp: Slot [6] registered [ 0.283291] acpiphp: Slot [3] registered [ 0.285079] acpiphp: Slot [4] registered [ 0.286126] acpiphp: Slot [7] registered [ 0.288109] acpiphp: Slot [8] registered [ 0.289097] acpiphp: Slot [9] registered [ 0.290104] acpiphp: Slot [10] registered [ 0.292116] acpiphp: Slot [11] registered [ 0.293103] acpiphp: Slot [12] registered [ 0.295170] acpiphp: Slot [13] registered [ 0.296128] acpiphp: Slot [14] registered [ 0.297123] acpiphp: Slot [15] registered [ 0.299103] acpiphp: Slot [16] registered [ 0.300109] acpiphp: Slot [17] registered [ 0.301106] acpiphp: Slot [18] registered [ 0.303135] acpiphp: Slot [19] registered [ 0.304106] acpiphp: Slot [20] registered [ 0.306106] acpiphp: Slot [21] registered [ 0.307105] acpiphp: Slot [22] registered [ 0.308115] acpiphp: Slot [23] registered [ 0.310132] acpiphp: Slot [24] registered [ 0.311090] acpiphp: Slot [25] registered [ 0.312113] acpiphp: Slot [26] registered [ 0.314157] acpiphp: Slot [27] registered [ 0.315104] acpiphp: Slot [28] registered [ 0.316086] acpiphp: Slot [29] registered [ 0.318100] acpiphp: Slot [30] registered [ 0.319104] acpiphp: Slot [31] registered [ 0.320096] PCI host bridge to bus 0000:00 [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340060] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.355112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366296] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369866] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375905] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379976] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.389985] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.394014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399890] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436087] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.464024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.474652] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.479610] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.483570] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.487608] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.491342] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.497243] iommu: Default domain type: Passthrough [ 0.499537] SCSI subsystem initialized [ 0.501147] ACPI: bus type USB registered [ 0.503141] usbcore: registered new interface driver usbfs [ 0.505094] usbcore: registered new interface driver hub [ 0.507093] usbcore: registered new device driver usb [ 0.509221] pps_core: LinuxPPS API ver. 1 registered [ 0.511015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.514105] PTP clock support registered [ 0.516244] EDAC MC: Ver: 3.0.0 [ 0.518256] PCI: Using ACPI for IRQ routing [ 0.520615] NetLabel: Initializing [ 0.522018] NetLabel: domain hash size = 128 [ 0.524016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.526114] NetLabel: unlabeled traffic allowed by default [ 0.528418] vgaarb: loaded [ 0.529334] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.532023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.543000] clocksource: Switched to clocksource kvm-clock [ 0.661317] VFS: Disk quotas dquot_6.6.0 [ 0.663747] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.667741] *** VALIDATE ramfs *** [ 0.669776] *** VALIDATE hugetlbfs *** [ 0.672150] pnp: PnP ACPI init [ 0.675888] pnp: PnP ACPI: found 6 devices [ 0.703085] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.707299] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.710195] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.712884] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.715023] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.718039] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.721445] NET: Registered protocol family 2 [ 0.724201] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.729360] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.733822] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.739398] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.743096] TCP: Hash tables configured (established 65536 bind 65536) [ 0.746532] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750072] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.753321] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.756750] NET: Registered protocol family 1 [ 0.759930] RPC: Registered named UNIX socket transport module. [ 0.763335] RPC: Registered udp transport module. [ 0.765474] RPC: Registered tcp transport module. [ 0.767364] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.770112] NET: Registered protocol family 44 [ 0.772226] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.774994] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.777331] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.779968] PCI: CLS 0 bytes, default 64 [ 0.782650] Unpacking initramfs... [ 2.190399] debug: unmapping init [mem 0xffff8e0d3cc64000-0xffff8e0d3ffcffff] [ 2.195203] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.197948] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.200971] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.706476] Initialise system trusted keyrings [ 2.708783] Key type blacklist registered [ 2.711360] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.721145] zbud: loaded [ 2.724528] *** VALIDATE nfs *** [ 2.725979] *** VALIDATE nfs4 *** [ 2.728242] pstore: using deflate compression [ 2.734322] Platform Keyring initialized [ 2.846624] NET: Registered protocol family 38 [ 2.848633] Key type asymmetric registered [ 2.850363] Asymmetric key parser 'x509' registered [ 2.852540] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.856230] io scheduler mq-deadline registered [ 2.858416] io scheduler kyber registered [ 2.860135] io scheduler bfq registered [ 2.862074] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.865525] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.868654] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.871084] ACPI: Power Button [PWRF] [ 2.876782] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.883405] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.894454] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.924124] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.951583] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.957864] Non-volatile memory driver v1.3 [ 2.959629] Linux agpgart interface v0.103 [ 3.001234] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.004721] vda: detected capacity change from 0 to 77111296 [ 3.023226] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.026674] vdb: detected capacity change from 0 to 1073741824 [ 3.041419] libphy: Fixed MDIO Bus: probed [ 3.047422] usbcore: registered new interface driver usbserial_generic [ 3.050809] usbserial: USB Serial support registered for generic [ 3.053394] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.058220] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.060500] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.062982] mousedev: PS/2 mouse device common for all mice [ 3.067833] rtc_cmos 00:05: RTC can wake from S4 [ 3.073081] rtc_cmos 00:05: registered as rtc0 [ 3.075217] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.075944] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.080261] intel_pstate: CPU model not supported [ 3.085285] hid: raw HID events driver (C) Jiri Kosina [ 3.092613] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.095923] usbcore: registered new interface driver usbhid [ 3.095934] usbhid: USB HID core driver [ 3.096113] drop_monitor: Initializing network drop monitor service [ 3.096270] Initializing XFRM netlink socket [ 3.096738] NET: Registered protocol family 10 [ 3.097937] Segment Routing with IPv6 [ 3.097977] NET: Registered protocol family 17 [ 3.098573] mpls_gso: MPLS GSO support [ 3.103919] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.128941] RAS: Correctable Errors collector initialized. [ 3.131985] AVX version of gcm_enc/dec engaged. [ 3.133860] AES CTR mode by8 optimization enabled [ 3.213809] sched_clock: Marking stable (3213768314, 0)->(4168396519, -954628205) [ 3.217708] registered taskstats version 1 [ 3.219929] Loading compiled-in X.509 certificates [ 3.222472] zswap: loaded using pool lzo/zbud [ 3.252379] Key type big_key registered [ 3.265583] Key type encrypted registered [ 3.267504] ima: No TPM chip found, activating TPM-bypass! [ 3.269936] ima: Allocated hash algorithm: sha1 [ 3.272110] ima: No architecture policies found [ 3.274193] evm: Initialising EVM extended attributes: [ 3.276310] evm: security.selinux [ 3.277877] evm: security.ima [ 3.279258] evm: security.capability [ 3.280791] evm: HMAC attrs: 0x1 [ 3.282923] rtc_cmos 00:05: setting system clock to 2026-09-29 19:39:25 UTC (1790710765) [ 3.290066] debug: unmapping init [mem 0xffffffff8dc03000-0xffffffff8ddfffff] [ 3.293677] debug: unmapping init [mem 0xffffffff8c982000-0xffffffff8cc58fff] [ 3.301123] Write protecting the kernel read-only data: 28672k [ 3.304762] debug: unmapping init [mem 0xffffffff8b003000-0xffffffff8b1fffff] [ 3.309094] debug: unmapping init [mem 0xffffffff8b914000-0xffffffff8b9fffff] [ 3.344874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.355912] systemd[1]: Detected virtualization kvm. [ 3.358167] systemd[1]: Detected architecture x86-64. [ 3.360220] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.388145] systemd[1]: No hostname configured. [ 3.391052] systemd[1]: Set hostname to . [ 3.393850] random: systemd: uninitialized urandom read (16 bytes read) [ 3.397315] systemd[1]: Initializing machine ID from random generator. [ 3.540396] random: systemd: uninitialized urandom read (16 bytes read) [ 3.544355] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.549452] random: systemd: uninitialized urandom read (16 bytes read) [ 3.552491] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.557580] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.140902] device-mapper: uevent: version 1.0.3 [ 4.143165] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.823324] virtio_net virtio0 ens2: renamed from eth0 [ 4.892475] scsi host0: ata_piix [ 4.953075] scsi host1: ata_piix [ 4.979755] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.982744] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.546953] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.789031] random: crng init done [ 9.790741] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.108472] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.267772] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.536378] SELinux: Disabled at runtime. [ 11.596069] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.607275] systemd[1]: Detected virtualization kvm. [ 11.609444] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.134278] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.137855] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.143098] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.147788] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.151370] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.160297] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.167341] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ 12.316856] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.595310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.941580] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.943796] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.041232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.059934] EDAC sbridge: Ver: 1.1.2 [ 14.157075] Key type dns_resolver registered [ 14.460357] NFS: Registering the id_resolver key type [ 14.463054] Key type id_resolver registered [ 14.464883] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg443-client login: [ 65.580342] libcfs: loading out-of-tree module taints kernel. [ 65.638345] Key type ._llcrypt registered [ 65.643972] Key type .llcrypt registered [ 66.119245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 66.145123] alg: No test for adler32 (adler32-zlib) [ 67.723554] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 68.559626] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 70.383218] Key type lgssc registered [ 72.433645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 198.997080] Lustre: Mounted lustre-client - version 2.17.59_2_g598166e [ 203.931426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.313338] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.735165] Lustre: lustre-OST0000-osc-ffff8e0d83eaf800: disconnect after 23s idle [ 225.874766] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing yml_node [ 230.295569] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 232.924944] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 235.406772] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 237.128823] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 29 15:43:17 EDT 2026 [ 240.464046] hrtimer: interrupt took 5916995 ns [ 252.595539] Lustre: Unmounted lustre-client [ 279.390793] Key type lgssc unregistered [ 279.701955] LNet: 8417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.710130] LNetError: 8417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.736853] LNet: Removed LNI 192.168.204.43@tcp [ 280.872296] Key type .llcrypt unregistered [ 280.874140] Key type ._llcrypt unregistered [ 310.685769] Key type ._llcrypt registered [ 310.687949] Key type .llcrypt registered [ 310.834847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 310.851846] alg: No test for adler32 (adler32-zlib) [ 311.723042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 311.818792] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 311.845447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 317.308580] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 317.401203] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.416554] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.451587] LNet: Removed LNI 192.168.204.43@tcp [ 318.369934] Key type .llcrypt unregistered [ 318.378556] Key type ._llcrypt unregistered [ 326.538617] Key type ._llcrypt registered [ 326.541106] Key type .llcrypt registered [ 328.137733] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:44:48 (1790711088) [ 328.582812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.609339] alg: No test for adler32 (adler32-zlib) [ 329.509623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.594093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.619239] LNet: 9718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.655768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.733650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.754653] LNet: 9725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 337.356522] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:44:58 (1790711098) [ 337.488849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.525088] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.554556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.688947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 344.751409] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:45:05 (1790711105) [ 344.969191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.003042] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.022313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.090890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 351.056024] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:45:12 (1790711112) [ 351.220739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.245053] LNet: 11475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.264508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.320979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 357.018543] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:45:17 (1790711117) [ 357.145656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.177229] LNet: 12060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.216699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.297177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 357.630522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 364.165426] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:45:25 (1790711125) [ 364.505934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.574805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.684943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 365.042752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 365.371922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 372.752513] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:45:33 (1790711133) [ 372.895737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.946231] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.957321] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 373.001785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.183221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 380.251325] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:45:41 (1790711141) [ 380.469699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.511710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.597552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 380.636471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.114354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 381.161405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 381.612908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 381.653242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.205666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 382.266853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 382.671375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 382.720584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 383.178604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 383.221364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.646978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 383.703737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 384.268650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 384.304745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 384.869575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 384.922267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 385.355868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 385.391272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 385.688138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 385.713882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 386.100366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 386.151605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.602456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.673749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 393.607163] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:45:54 (1790711154) [ 393.801843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.824789] LNet: 14516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.830262] LNet: 14516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 393.874137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.001641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 394.043917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 399.773970] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:46:00 (1790711160) [ 399.923754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.998634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.193617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.303653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 406.497338] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:46:07 (1790711167) [ 406.627928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.685382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.750729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 406.777252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 412.556714] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:46:13 (1790711173) [ 412.634466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.680666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.773252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 412.804740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 418.718754] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:46:19 (1790711179) [ 418.937808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.996584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.118985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 419.171757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 425.353576] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:46:26 (1790711186) [ 425.658367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.689484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.773318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 425.822550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 431.755300] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:46:32 (1790711192) [ 431.969943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.005081] LNet: 18031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.011232] LNet: 18031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 432.054388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.129952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.484088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.259832] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:46:39 (1790711199) [ 438.514422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.543197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.754578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.953465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.149426] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:46:45 (1790711205) [ 445.358225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.383235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.466807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.631429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.152946] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:46:52 (1790711212) [ 452.314982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.376315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.941981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.185381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.187853] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:47:00 (1790711220) [ 460.390226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.452224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.537325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.608868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.131663] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:47:08 (1790711228) [ 468.365281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.447766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.607541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.698383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.061407] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:47:17 (1790711237) [ 477.262687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.299640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.386925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 484.229438] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:47:25 (1790711245) [ 484.327655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.387664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.483560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 491.343653] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:47:32 (1790711252) [ 491.464853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.511590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.597848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 492.021390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 500.250913] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:47:40 (1790711260) [ 500.356198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.384821] LNet: 23466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.390113] LNet: 23466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 500.416454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.507710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 500.546423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 507.169073] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:47:48 (1790711268) [ 507.471280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.509229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.636432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 507.677923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 515.259477] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:47:56 (1790711276) [ 515.577725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.654507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.766287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 515.833849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 523.246781] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:48:04 (1790711284) [ 523.689916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.743423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.803099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 523.837515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 530.590983] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:48:11 (1790711291) [ 530.742383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.795842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.877658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 530.937455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 530.956354] LNetError: 25829:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 531.005200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 531.039612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 536.735924] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:48:17 (1790711297) [ 536.805476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.837038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.935700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6qDGMjtICW [ 537.002636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6qDGMjtICW [ 537.081105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6qDGMjtICW [ 542.580657] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:48:23 (1790711303) [ 542.711506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.752733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.800664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 542.825635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 542.864426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 548.132407] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:48:29 (1790711309) [ 548.222994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.268345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.368345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 548.400336] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 549.507830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 550.519978] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 552.383266] Key type lgssc registered [ 553.458471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.620114] LNetError: 27590:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.43@tcp99 has no available nets [ 558.835262] Key type lgssc unregistered [ 559.111770] LNetError: 28703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.126986] LNet: Removed LNI 192.168.204.43@tcp [ 559.566572] Key type .llcrypt unregistered [ 559.570320] Key type ._llcrypt unregistered [ 564.730864] Key type ._llcrypt registered [ 564.734314] Key type .llcrypt registered [ 565.902857] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:48:46 (1790711326) [ 566.412288] Key type .llcrypt unregistered [ 566.416178] Key type ._llcrypt unregistered [ 573.024559] Key type ._llcrypt registered [ 573.026691] Key type .llcrypt registered [ 573.228613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 573.256730] alg: No test for adler32 (adler32-zlib) [ 574.154430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.198938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 574.219892] LNet: 29624:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 574.232113] LNet: 29624:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 574.240170] LNet: 29624:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 574.251333] LNet: Message forwarding will be enabled in 2s [ 577.311178] LNet: Message forwarding enabled [ 577.717841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 577.751740] LNet: Message forwarding will stop in 2s to 6s [ 581.409598] LNet: Message forwarding disabled [ 582.277914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 582.295570] LNet: 29722:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 582.305425] LNet: 29722:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 582.318517] LNet: 29722:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 582.334794] LNet: Message forwarding will be enabled in 2s [ 582.500339] LNet: 29735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.161675] Key type .llcrypt unregistered [ 583.166622] Key type ._llcrypt unregistered [ 594.815919] Key type ._llcrypt registered [ 594.818440] Key type .llcrypt registered [ 596.222074] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:49:16 (1790711356) [ 596.738288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.757618] alg: No test for adler32 (adler32-zlib) [ 597.763478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.878088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 604.628189] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:49:25 (1790711365) [ 604.740401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.760846] LNet: 31133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.796289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.867646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 604.903432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 604.934985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 604.972917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 605.000955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 605.034655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 605.059791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 605.087570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 605.116227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 605.138445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 605.173450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 605.198794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 605.218246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 605.245388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 605.289480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 605.330568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 605.373183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 605.419812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 605.458027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 605.493962] LNetError: 31180:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 611.177358] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:49:32 (1790711372) [ 611.336541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.355972] LNet: 31746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.389289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.512244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.742544] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:49:39 (1790711379) [ 620.556491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg443-server [ 622.493572] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:49:43 (1790711383) [ 622.594094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.637661] LNet: 32676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.664259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.765556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.811417] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 623.680104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[44-48]@tcp [ 623.710674] LNet: 32815:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 623.758129] LNetError: 32686:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 623.778121] LNetError: 32686:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.44@tcp has gone from up to down [ 631.037864] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:49:51 (1790711391) [ 632.915206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg443-server [ 634.870942] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:49:55 (1790711395) [ 635.116713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.148051] LNet: 33750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.165753] LNetError: 33750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.194843] LNet: Removed LNI 192.168.204.43@tcp [ 635.272762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.362131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.411509] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 636.198140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 636.261557] LNet: 33868:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 636.279480] LNet: 33868:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 636.308766] LNetError: 33755:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 636.327551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 636.336758] LNetError: 33755:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 636.362467] LNetError: 33755:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.44@tcp has gone from up to down [ 636.377836] LNetError: 33755:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 642.400110] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:50:03 (1790711403) [ 642.491836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.512679] LNet: 34455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.518669] LNetError: 34455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.562739] LNet: Removed LNI 192.168.204.43@tcp [ 643.618804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.770255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 643.891340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.948845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.009419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 644.057080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.085279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.143360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 644.226244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.283439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.372194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 644.471684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.529410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.661952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 644.777984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.826300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.911668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.028264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.101773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.173729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 645.236980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.264613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.346235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.448394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.487381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.553405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 645.650333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.735937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.825632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.917197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.961365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.093479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 646.186293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.233995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.325995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.642772] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:50:13 (1790711413) [ 654.408977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg443-server [ 656.398524] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:50:17 (1790711417) [ 658.237753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg443-server [ 660.024952] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:50:20 (1790711420) [ 660.204879] LNet: 35860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.217305] LNet: 35860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 660.916101] Key type .llcrypt unregistered [ 660.922040] Key type ._llcrypt unregistered [ 666.618559] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.861402] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.875638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.893234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.578577] Key type ._llcrypt registered [ 669.586177] Key type .llcrypt registered [ 669.929841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.958137] alg: No test for adler32 (adler32-zlib) [ 671.031943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.096961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.130416] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 671.881471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 671.903793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.559414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 674.589554] LNet: Removed LNI 10.1.2.3@tcp [ 674.779788] LNet: 36580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.798671] LNetError: 36580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.825477] LNet: Removed LNI 192.168.204.43@tcp [ 675.583551] Key type .llcrypt unregistered [ 675.588867] Key type ._llcrypt unregistered [ 688.205902] Key type ._llcrypt registered [ 688.209652] Key type .llcrypt registered [ 689.549679] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:50:50 (1790711450) [ 690.157480] Key type .llcrypt unregistered [ 690.160555] Key type ._llcrypt unregistered [ 696.202865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 696.409895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 696.426038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 696.436472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 698.962493] Key type ._llcrypt registered [ 698.965503] Key type .llcrypt registered [ 699.122603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.134165] alg: No test for adler32 (adler32-zlib) [ 700.064919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.144101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.167400] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 700.811487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 700.833479] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.275968] LNet: 38088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.288799] LNetError: 38088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 703.298125] LNet: Removed LNI 10.1.2.3@tcp [ 703.312078] LNet: Removed LNI 192.168.204.43@tcp [ 704.081386] Key type .llcrypt unregistered [ 704.085342] Key type ._llcrypt unregistered [ 715.845201] Key type ._llcrypt registered [ 715.847783] Key type .llcrypt registered [ 717.124217] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:51:17 (1790711477) [ 717.872294] Key type .llcrypt unregistered [ 717.873766] Key type ._llcrypt unregistered [ 724.732744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.931589] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.976509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.995707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.624238] Key type ._llcrypt registered [ 727.629713] Key type .llcrypt registered [ 727.938041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.963890] alg: No test for adler32 (adler32-zlib) [ 728.982537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.077186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 729.113527] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 731.128152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 731.176691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.398331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 733.437769] LNet: Removed LNI 10.1.2.3@tcp [ 733.452496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 733.472988] LNet: Removed LNI 10.1.2.31@tcp [ 733.487697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 733.522378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.236817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 734.263408] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 735.432513] LNet: 40004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.441830] LNetError: 40004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 735.447567] LNet: Removed LNI 10.1.2.31@tcp [ 735.462626] LNet: Removed LNI 10.1.2.3@tcp [ 735.982749] Key type .llcrypt unregistered [ 735.989722] Key type ._llcrypt unregistered [ 749.892910] Key type ._llcrypt registered [ 749.898402] Key type .llcrypt registered [ 751.307615] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:51:51 (1790711511) [ 751.844397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.860942] alg: No test for adler32 (adler32-zlib) [ 752.806669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.882911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.915176] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 753.746994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.765274] LNet: 40957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.772872] LNetError: 40957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.788622] LNet: Removed LNI 192.168.204.43@tcp [ 753.810035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.856139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.43@tcp [ 753.890730] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 754.803188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.43@tcp [ 754.833243] LNet: Removed LNI 192.168.204.43@tcp [ 754.909831] LNet: 41092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.762617] Key type .llcrypt unregistered [ 755.765628] Key type ._llcrypt unregistered [ 767.699401] Key type ._llcrypt registered [ 767.705409] Key type .llcrypt registered [ 769.401865] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:52:09 (1790711529) [ 771.321723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg443-server [ 773.132885] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:52:13 (1790711533) [ 773.636378] Key type .llcrypt unregistered [ 773.647433] Key type ._llcrypt unregistered [ 778.829831] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.053711] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.060143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.066861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.612720] Key type ._llcrypt registered [ 781.614586] Key type .llcrypt registered [ 781.840630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.855855] alg: No test for adler32 (adler32-zlib) [ 782.762714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.826547] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 783.367545] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 785.818649] LNet: 42929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.836912] LNetError: 42929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.845819] LNet: Removed LNI 192.168.204.43@tcp1 [ 785.865402] LNet: Removed LNI 10.1.2.3@tcp2 [ 786.796371] Key type .llcrypt unregistered [ 786.805624] Key type ._llcrypt unregistered [ 799.587340] Key type ._llcrypt registered [ 799.591779] Key type .llcrypt registered [ 801.404935] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:52:41 (1790711561) [ 801.856768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.884519] alg: No test for adler32 (adler32-zlib) [ 802.766406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.814374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.845237] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 803.567593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.581418] LNet: 43899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.587663] LNetError: 43899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.599638] LNet: Removed LNI 192.168.204.43@tcp [ 803.630167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.687151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 803.705303] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 811.008307] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:52:51 (1790711571) [ 811.154305] LNet: 44484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.158817] LNetError: 44484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.200663] LNet: Removed LNI 192.168.204.43@tcp [ 811.971760] Key type .llcrypt unregistered [ 811.974995] Key type ._llcrypt unregistered [ 818.211853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.418823] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.427415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.431503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.832933] Key type ._llcrypt registered [ 820.835324] Key type .llcrypt registered [ 820.992729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.017751] alg: No test for adler32 (adler32-zlib) [ 821.923514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.970897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.000075] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 822.572432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.595918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.323400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.346620] LNet: 45233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.355375] LNetError: 45233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.366417] LNet: Removed LNI 10.1.2.3@tcp [ 824.375859] LNet: Removed LNI 192.168.204.43@tcp [ 824.405565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.497635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.522033] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 824.526532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.833320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.850291] LNet: 45260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.857326] LNetError: 45260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.864151] LNetError: 45260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.873550] LNet: Removed LNI 10.1.2.3@tcp [ 824.886938] LNet: Removed LNI 192.168.204.43@tcp [ 824.919337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.966078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.995514] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 825.001642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.286879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 825.302039] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 826.166640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.180590] LNet: 45422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.185785] LNetError: 45422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 826.190455] LNetError: 45422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 826.195518] LNet: Removed LNI 10.1.2.3@tcp [ 826.198349] LNet: Removed LNI 192.168.204.43@tcp [ 826.208477] LNet: Removed LNI 192.168.204.43@tcp2 [ 826.224617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.278522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 826.310474] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 826.314091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.317531] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 832.320749] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:53:13 (1790711593) [ 832.595798] LNet: 46019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.602437] LNetError: 46019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.609150] LNetError: 46019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 832.620382] LNet: Removed LNI 10.1.2.3@tcp [ 832.624448] LNet: Removed LNI 192.168.204.43@tcp [ 832.645528] LNet: Removed LNI 192.168.204.43@tcp2 [ 833.252173] Key type .llcrypt unregistered [ 833.255650] Key type ._llcrypt unregistered [ 838.257609] Key type ._llcrypt registered [ 838.259461] Key type .llcrypt registered [ 838.412930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.425243] alg: No test for adler32 (adler32-zlib) [ 839.304792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.349353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.370648] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 839.966585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.985399] LNet: 46412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.993303] LNetError: 46412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.011362] LNet: Removed LNI 192.168.204.43@tcp [ 840.028202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.076817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.098623] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 840.429911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.462223] LNet: Removed LNI 192.168.204.43@tcp [ 840.477751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.526791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.552720] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 840.905278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.923530] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.934693] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 840.941428] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.948196] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.961480] LNet: Removed LNI 192.168.204.43@tcp [ 840.979822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.040352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.068651] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 841.496621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.657169] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.661526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.666616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.956314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.974439] LNet: 46551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.979188] LNetError: 46551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.992645] LNet: Removed LNI 192.168.204.43@tcp [ 844.018853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.081674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.103940] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 850.000149] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:53:30 (1790711610) [ 850.132226] LNet: 47141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.143132] LNetError: 47141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.160176] LNet: Removed LNI 192.168.204.43@tcp [ 850.885193] Key type .llcrypt unregistered [ 850.894511] Key type ._llcrypt unregistered [ 856.885576] Key type ._llcrypt registered [ 856.888058] Key type .llcrypt registered [ 857.118743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.141973] alg: No test for adler32 (adler32-zlib) [ 858.114733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.244767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.274545] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 859.396223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.427714] LNet: 47545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.432983] LNetError: 47545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.450953] LNet: Removed LNI 192.168.204.43@tcp [ 859.468158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.536829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 859.566833] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 866.432189] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:53:47 (1790711627) [ 866.802586] LNet: 48137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.815802] LNetError: 48137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.837205] LNet: Removed LNI 192.168.204.43@tcp [ 867.548707] Key type .llcrypt unregistered [ 867.553632] Key type ._llcrypt unregistered [ 873.311029] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.529299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.543275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.550965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 876.143557] Key type ._llcrypt registered [ 876.150904] Key type .llcrypt registered [ 876.363431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.376362] alg: No test for adler32 (adler32-zlib) [ 877.291862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.347355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.369532] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 878.270146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.287421] LNet: 48585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.294732] LNetError: 48585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.312792] LNet: Removed LNI 192.168.204.43@tcp [ 878.343276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.420829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 878.459099] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 885.059888] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:54:05 (1790711645) [ 885.211518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.284912] LNet: 49179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.302909] LNetError: 49179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.335285] LNet: Removed LNI 192.168.204.43@tcp [ 885.400966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.551755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 894.321716] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:54:14 (1790711654) [ 894.440749] LNet: 49757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.382515] Key type .llcrypt unregistered [ 895.385605] Key type ._llcrypt unregistered [ 903.099522] Key type ._llcrypt registered [ 903.101237] Key type .llcrypt registered [ 903.454912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.476877] alg: No test for adler32 (adler32-zlib) [ 904.632306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.721707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.783941] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 905.826139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.852478] LNet: 50166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.858947] LNetError: 50166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.874733] LNet: Removed LNI 192.168.204.43@tcp [ 905.904983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.988406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 906.029876] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 914.150606] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:54:34 (1790711674) [ 914.304912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.326453] LNet: 50752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.347768] LNetError: 50752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.380354] LNet: Removed LNI 192.168.204.43@tcp [ 914.414499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.526822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 922.453861] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:54:42 (1790711682) [ 924.594311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 927.219510] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:54:47 (1790711687) [ 929.021893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 931.623067] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:54:51 (1790711691) [ 934.321359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 936.843701] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:54:56 (1790711696) [ 938.698432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 940.470581] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:55:01 (1790711701) [ 942.415868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 944.680235] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:55:04 (1790711704) [ 944.826650] LNet: 52773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.675639] Key type .llcrypt unregistered [ 945.680751] Key type ._llcrypt unregistered [ 952.958392] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.217453] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.228056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.237685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.816854] Key type ._llcrypt registered [ 955.821331] Key type .llcrypt registered [ 956.040618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.070460] alg: No test for adler32 (adler32-zlib) [ 957.151126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.248626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.284419] LNet: Added LNI 192.168.204.43@tcp [8/512/0/180] [ 957.293647] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 957.298300] LNet: 53101:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.395910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.414439] LNet: 53120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.419574] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.427595] LNet: Removed LNI 10.1.2.3@tcp [ 957.439598] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.458348] LNet: Removed LNI 192.168.204.43@tcp [ 957.527363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.676147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.713200] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 957.851380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.882812] LNet: 53144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.888497] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.915389] LNet: Removed LNI 192.168.204.43@tcp [ 957.965778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.096314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.120311] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 958.125473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 958.128055] LNet: 53152:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 958.237967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.258753] LNet: 53170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.266822] LNetError: 53170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.272793] LNet: Removed LNI 10.1.2.3@tcp [ 958.274041] LNetError: 53170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.296750] LNet: Removed LNI 192.168.204.43@tcp [ 958.339947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.427586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.464548] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 958.471886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 958.475770] LNet: 53178:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 958.942984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.031408] LNet: 53204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.048735] LNetError: 53204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 959.066316] LNet: Removed LNI 10.1.2.3@tcp [ 959.086620] LNetError: 53204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.108485] LNet: Removed LNI 192.168.204.43@tcp [ 959.165397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.288659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 959.311699] LNet: Added LNI 192.168.204.43@tcp [8/256/0/182] [ 959.318666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 959.434438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.450812] LNet: 53232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.457788] LNetError: 53232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 959.466466] LNet: Removed LNI 10.1.2.3@tcp [ 959.469771] LNetError: 53232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.481313] LNet: Removed LNI 192.168.204.43@tcp [ 959.499146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.570572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 959.595269] LNet: Added LNI 192.168.204.43@tcp [8/256/0/182] [ 959.600176] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 959.603418] LNet: 53240:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 959.675502] LNet: Removed LNI 10.1.2.3@tcp [ 959.688096] LNet: Removed LNI 192.168.204.43@tcp [ 960.787104] Key type .llcrypt unregistered [ 960.792730] Key type ._llcrypt unregistered [ 974.832677] Key type ._llcrypt registered [ 974.843307] Key type .llcrypt registered [ 977.497834] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:55:36 (1790711736) [ 980.349680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 983.176901] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:55:43 (1790711743) [ 983.838923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.856922] alg: No test for adler32 (adler32-zlib) [ 984.943634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.035026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.081224] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 986.312767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.43 [ 986.337527] LNet: 54505:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 986.346868] LNet: Removed LNI 192.168.204.43@tcp [ 994.920931] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:55:55 (1790711755) [ 995.066211] LNet: 55068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.081869] LNetError: 55068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.108844] LNet: Removed LNI 192.168.204.43@tcp [ 996.210425] Key type .llcrypt unregistered [ 996.215672] Key type ._llcrypt unregistered [ 1003.291070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1003.667314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1006.409451] Key type ._llcrypt registered [ 1006.414548] Key type .llcrypt registered [ 1006.705406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.738489] alg: No test for adler32 (adler32-zlib) [ 1008.148253] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.024624] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:56:15 (1790711775) [ 1015.547337] LNet: 55988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.557836] LNetError: 55988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.579927] LNet: Removed LNI 10.1.2.3@tcp [ 1016.540348] Key type .llcrypt unregistered [ 1016.543749] Key type ._llcrypt unregistered [ 1024.137941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.493211] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1024.499150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1024.503359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1027.197905] Key type ._llcrypt registered [ 1027.200533] Key type .llcrypt registered [ 1027.461491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.489820] alg: No test for adler32 (adler32-zlib) [ 1028.426384] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.875861] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:56:36 (1790711796) [ 1037.927801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1040.515850] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:56:40 (1790711800) [ 1040.829663] LNet: 57172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.845540] LNetError: 57172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.862531] LNet: Removed LNI 10.1.2.3@tcp [ 1041.593115] Key type .llcrypt unregistered [ 1041.600287] Key type ._llcrypt unregistered [ 1048.578412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1048.950567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1048.956393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1048.962975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1051.914438] Key type ._llcrypt registered [ 1051.916724] Key type .llcrypt registered [ 1052.154206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.178622] alg: No test for adler32 (adler32-zlib) [ 1053.225877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.342435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.435341] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:57:03 (1790711823) [ 1064.205483] LNet: 58401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.216140] LNetError: 58401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.232396] LNet: Removed LNI 10.1.2.3@tcp [ 1065.265555] Key type .llcrypt unregistered [ 1065.268990] Key type ._llcrypt unregistered [ 1071.843732] Key type ._llcrypt registered [ 1071.845957] Key type .llcrypt registered [ 1072.022275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.050529] alg: No test for adler32 (adler32-zlib) [ 1073.028490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1073.055384] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1082.013613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1082.921673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1083.007764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1083.061537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1083.093994] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1083.301564] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1083.340839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.383191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.438530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.475650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.521856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1083.585805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.630790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.678634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.716743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.738143] LNet: 58840:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.756277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.807564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.843471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.886956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.917375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.953920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.002351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.701669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1084.789704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1084.848837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1084.868784] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1085.094808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.144983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.198799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.236794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.284720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1085.352895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.381759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.419955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.453372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.465613] LNet: 58952:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.480102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.508750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.536614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.574925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.606778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.636918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.681745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.348563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1086.388178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1086.465658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1086.497552] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1086.504322] LNet: Skipped 1 previous similar message [ 1086.749989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.782994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1086.808579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.833634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.854904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1086.883069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.908335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.955430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.983639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.007027] LNet: 59063:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.040408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.081917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.123618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.184642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.211382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.235176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.264917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1087.837680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1087.913387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1087.983691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1088.242447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.278980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.330656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.366525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.403519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1088.444297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.492972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.540266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.599602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.633908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.671877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.723512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.766294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.809589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.885990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.963241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1089.524579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1089.618169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1089.678546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1089.704570] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1089.717651] LNet: Skipped 1 previous similar message [ 1089.994060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.032945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.064298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.113689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.186295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1090.255417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.306067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.377286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.430781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.463729] LNet: 59286:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.469756] LNet: 59286:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1090.500220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.544827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.592871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.668502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.722792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.757547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.795161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.254310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1091.321310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1091.377922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1091.597859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.627702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.665213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.702985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.754955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1091.821939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.866937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.910978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.984515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.041886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.083968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.141878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.202647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.256637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.286811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.329649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.675653] LNet: 59438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.683211] LNetError: 59438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.740795] LNet: Removed LNI 192.168.204.43@tcp [ 1094.177695] Key type .llcrypt unregistered [ 1094.183604] Key type ._llcrypt unregistered [ 1110.578720] Key type ._llcrypt registered [ 1110.580878] Key type .llcrypt registered [ 1112.154741] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:57:52 (1790711872) [ 1113.041661] Key type .llcrypt unregistered [ 1113.046134] Key type ._llcrypt unregistered [ 1119.772160] Key type ._llcrypt registered [ 1119.774368] Key type .llcrypt registered [ 1120.060859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.087865] alg: No test for adler32 (adler32-zlib) [ 1121.091345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.125592] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1129.914813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1142.966449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1143.012504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1143.065106] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1143.769412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1144.512514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1144.571948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 1144.626704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1144.686588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 1144.729367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1144.761937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 1144.812374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1144.852348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 1144.920481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1144.955817] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1144.965326] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp1 [ 1145.361457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.437936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.519258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.546570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.593806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1145.630684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.677715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.736976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.760123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.777557] LNet: 60919:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.788673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.816791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.869897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.916481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.955594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.006034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.024262] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1146.048771] LNet: Skipped 3 previous similar messages [ 1146.076203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.855119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1146.925748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 1146.986504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1147.077895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 1147.122461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1147.192807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 1147.267881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1147.347319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 1147.430485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1147.467748] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1147.833394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.867840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.913041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.953549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.031643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1148.094323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.138415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.181547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.217199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.240310] LNet: 61043:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.275254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.324854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.362529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.423755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.463950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.491911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.523347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.014936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1149.062681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 1149.099318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1149.135276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 1149.171762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1149.208488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 1149.238942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1149.291125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 1149.354702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1149.688080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.728765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.766381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.796889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.867276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1149.924567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.977328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.023740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.070782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.097305] LNet: 61166:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.130058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.163977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.202385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.252731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.283152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.315512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.348605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1151.064651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1151.141033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 1151.205604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1151.260769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 1151.303330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1151.346254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 1151.376502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1151.411938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 1151.466257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1151.503552] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1151.518309] LNet: Skipped 5 previous similar messages [ 1151.811810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.848818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.899136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.957300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.007687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1152.060150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.107241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.145708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.227784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.253375] LNet: 61289:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.274947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.317307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.351254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.435883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.493807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.558338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.619141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1153.437844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1153.491411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 1153.553363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1153.640109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 1153.677160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1153.729578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 1153.776950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1153.823844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 1153.894805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1154.400403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.433434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.477360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.518645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.561396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1154.631599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.692039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.731277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.782715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.853395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.894842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.922450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.967778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.045289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.109409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.181717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1155.959138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1156.005689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 1156.072470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1156.150876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 1156.184067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1156.241848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 1156.274995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1156.321624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 1156.380985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1156.408601] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1156.415189] LNet: Skipped 5 previous similar messages [ 1156.680540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.721938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.771659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.838761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.882984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1156.921579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.007707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.096406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.150916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.166506] LNet: 61536:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.179271] LNet: 61536:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1157.208357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.245431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.272623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.331845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.364461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.427943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.483664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.585449] LNet: 61612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.602423] LNetError: 61612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.624134] LNet: Removed LNI 192.168.204.43@tcp [ 1163.688184] LNet: Removed LNI 192.168.204.43@tcp1 [ 1164.370992] Key type .llcrypt unregistered [ 1164.377178] Key type ._llcrypt unregistered [ 1183.303752] Key type ._llcrypt registered [ 1183.320451] Key type .llcrypt registered [ 1184.788835] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:59:05 (1790711945) [ 1185.901476] Key type .llcrypt unregistered [ 1185.905623] Key type ._llcrypt unregistered [ 1193.336061] Key type ._llcrypt registered [ 1193.337869] Key type .llcrypt registered [ 1193.604865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1193.620280] alg: No test for adler32 (adler32-zlib) [ 1194.694644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1194.731277] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1203.883716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1204.857658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1204.901213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1204.959355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1204.987616] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1205.256294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.281850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.308607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.337596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.363082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1205.390424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.416767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.447366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.474444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.489330] LNet: 62863:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.511021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.544969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.577303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.603810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.647510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.697746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.760264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.443347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1206.496698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1206.544884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1206.575884] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1206.893913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.939580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.040209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.102815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.147673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1207.180619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.257045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.338170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.409145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.445388] LNet: 62975:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.465644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.506384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.577656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.623654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.705715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.766473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.833626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1208.437259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1208.502967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1208.538092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1208.560923] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1208.867691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.897775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.926989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.960164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.014205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1209.086932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.165043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.223248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.297710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.347133] LNet: 63086:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.393379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.470905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.534826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.625346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.698605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.765603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.833343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.250445] LNet: 63127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.260441] LNetError: 63127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.295862] LNet: Removed LNI 192.168.204.43@tcp [ 1210.901619] Key type .llcrypt unregistered [ 1210.910367] Key type ._llcrypt unregistered [ 1230.727574] Key type ._llcrypt registered [ 1230.730702] Key type .llcrypt registered [ 1232.030430] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:59:52 (1790711992) [ 1233.074482] Key type .llcrypt unregistered [ 1233.078764] Key type ._llcrypt unregistered [ 1239.953454] Key type ._llcrypt registered [ 1239.955717] Key type .llcrypt registered [ 1240.160265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.201394] alg: No test for adler32 (adler32-zlib) [ 1241.150524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1241.174459] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1249.190183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1259.714416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.811546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1259.854287] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1260.979796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1261.795598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1261.893949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 1261.951526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1262.019562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 1262.092486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1262.147693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 1262.176599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1262.216322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 1262.282837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1262.308302] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1262.315910] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp1 [ 1262.576617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.639900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.743611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.781641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.876996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1262.954310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.980101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.007190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.057482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.091555] LNet: 64602:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.106372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.130747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.162095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.192227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.223228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.265311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.318683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.351485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.568458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1264.669288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 1264.775042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1264.870592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 1265.006495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1265.085374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 1265.163334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1265.239586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 1265.344592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1265.414298] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1265.429913] LNet: Skipped 2 previous similar messages [ 1265.905331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.954237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1266.038381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.089389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.159935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1266.221660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.304642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.381894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.457784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.483520] LNet: 64728:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.541703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.595115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.680649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.756208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.821986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.884053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.039108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.117982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.162335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1268.223899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 1268.268987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1268.328915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 1268.397364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1268.483797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 1268.564420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1268.698566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 1268.771627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1268.799455] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1268.808979] LNet: Skipped 1 previous similar message [ 1269.267762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.374982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.461477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.518489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.580329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1269.630831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.690116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.765806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.864217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.896158] LNet: 64854:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.943480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.021316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.072188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.156507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.222704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.308849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.372541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.469826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.532287] LNet: 64932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.541976] LNetError: 64932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.553627] LNet: Removed LNI 192.168.204.43@tcp [ 1277.568803] LNet: Removed LNI 192.168.204.43@tcp1 [ 1278.300968] Key type .llcrypt unregistered [ 1278.308447] Key type ._llcrypt unregistered [ 1295.370417] Key type ._llcrypt registered [ 1295.373620] Key type .llcrypt registered [ 1297.251817] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:00:57 (1790712057) [ 1299.355708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1302.085156] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:01:02 (1790712062) [ 1303.732434] Key type .llcrypt unregistered [ 1303.735460] Key type ._llcrypt unregistered [ 1312.670094] Key type ._llcrypt registered [ 1312.673513] Key type .llcrypt registered [ 1313.255920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1313.300426] alg: No test for adler32 (adler32-zlib) [ 1314.413153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1314.442941] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1324.197853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1325.558538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1325.633584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1325.711841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1325.755412] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1326.164979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.213377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.272187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.328395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.380917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1326.477661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.542390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.568418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.610364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.629631] LNet: 66494:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1326.648890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1326.680084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1326.724858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.755534] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1326.761178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1326.769259] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1326.801795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1326.830926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.096895] LNet: 66533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.101279] LNetError: 66533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.167930] LNet: Removed LNI 192.168.204.43@tcp [ 1328.587563] Key type .llcrypt unregistered [ 1328.592787] Key type ._llcrypt unregistered [ 1346.674194] Key type ._llcrypt registered [ 1346.676913] Key type .llcrypt registered [ 1346.896668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1346.923408] alg: No test for adler32 (adler32-zlib) [ 1347.876757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1347.933898] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1355.693370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1365.943616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.006800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.054542] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1366.961681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1367.610788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1367.644059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 1367.677274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1367.717771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 1367.769030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1367.811129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 1367.855865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1367.902979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 1367.973775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1368.010330] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1368.317123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1368.349274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1368.383559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1368.416446] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp1 [ 1368.430064] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp1 [ 1368.449230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1368.530811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1368.609921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1368.694911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1368.745782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1368.797533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1368.815444] LNet: 67421:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.836896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.874327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.903406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.940781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.968454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.003887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1373.784104] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1374.192819] LNet: 67495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.210396] LNetError: 67495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.220097] LNet: Removed LNI 192.168.204.43@tcp [ 1374.239957] LNet: Removed LNI 192.168.204.43@tcp1 [ 1374.698947] Key type .llcrypt unregistered [ 1374.706036] Key type ._llcrypt unregistered [ 1393.576369] Key type ._llcrypt registered [ 1393.578466] Key type .llcrypt registered [ 1395.023619] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:02:35 (1790712155) [ 1395.389332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.399280] alg: No test for adler32 (adler32-zlib) [ 1396.342948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.432502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.469985] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1397.318717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.343318] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1398.366780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1398.436494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1398.473982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1398.509260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1398.565202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1398.648825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1398.715366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 1398.757503] LNetError: 68596:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1399.455518] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1399.455573] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1401.504792] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1402.376106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1405.599496] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1406.049390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1406.089882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1406.152779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.183080] LNet: 68703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.191007] LNetError: 68703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.200557] LNet: Removed LNI 192.168.204.43@tcp [ 1406.210572] LNet: Removed LNI 192.168.204.43@tcp1 [ 1406.245829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.350588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.395769] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1407.282110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.317647] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1408.227981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1408.263041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.314406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1408.350404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1408.380360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1408.407993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1408.439694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1408.486728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 1408.510550] LNetError: 68976:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1409.375770] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1412.134890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1415.519490] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1415.525341] LNet: Skipped 1 previous similar message [ 1420.470365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1420.505827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1420.549491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1427.527938] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:03:08 (1790712188) [ 1427.776231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.802172] LNet: 69694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.816736] LNetError: 69694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.823636] LNetError: 69694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1427.829536] LNet: Removed LNI 192.168.204.43@tcp [ 1427.842184] LNet: Removed LNI 192.168.204.43@tcp1 [ 1427.874083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.940918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1427.967788] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1428.576440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1428.602604] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1429.470369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1429.521494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1429.556610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1429.611433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1429.674190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1429.714445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1429.765143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 1429.861974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1429.983185] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.43@tcp [ 1431.015507] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1431.022322] LNet: Skipped 1 previous similar message [ 1445.527379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1445.784291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1445.838721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.896798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 500 [ 1446.065866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.843643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.915914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.945566] LNet: 70178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.952385] LNetError: 70178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.958608] LNetError: 70178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1451.965946] LNet: Removed LNI 192.168.204.43@tcp [ 1451.988883] LNet: Removed LNI 192.168.204.43@tcp1 [ 1452.023756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.104814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.140716] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1452.895188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.910986] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1453.810239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1453.875973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.978487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1454.044794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1454.117519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1454.171612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1454.216958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1454.260745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 1455.135685] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1455.148649] LNet: Skipped 3 previous similar messages [ 1455.662843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1466.539384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1466.603674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1466.647409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1475.372965] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:03:56 (1790712236) [ 1476.188586] LNet: 71225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.199574] LNetError: 71225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.206242] LNetError: 71225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1476.219936] LNet: Removed LNI 192.168.204.43@tcp [ 1476.241756] LNet: Removed LNI 192.168.204.43@tcp1 [ 1477.217953] Key type .llcrypt unregistered [ 1477.220655] Key type ._llcrypt unregistered [ 1485.788207] Key type ._llcrypt registered [ 1485.795250] Key type .llcrypt registered [ 1486.030976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1486.051995] alg: No test for adler32 (adler32-zlib) [ 1487.151968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1487.207574] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1496.594521] Lustre: DEBUG MARKER: Initial discovery [ 1496.831911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1499.320795] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1499.412623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1499.460025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1499.507300] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1499.539480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1499.587591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -l 3 [ 1499.640275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -e local_error [ 1499.684048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m PUT -l 6 [ 1500.511677] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1501.064720] Lustre: DEBUG MARKER: Force oleg443-client.virtnet to discover 192.168.204.143@tcp (in background) [ 1501.393921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1503.107744] Lustre: DEBUG MARKER: Force oleg443-server to discover 192.168.204.43@tcp [ 1504.480532] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1505.715085] Lustre: DEBUG MARKER: Wait for 71989 [ 1507.388997] Lustre: DEBUG MARKER: Finished wait on 71989 [ 1507.439576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1507.509313] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1507.535495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1507.979732] LNet: 72438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.994810] LNetError: 72438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.011399] LNet: Removed LNI 192.168.204.43@tcp [ 1508.592612] Key type .llcrypt unregistered [ 1508.595572] Key type ._llcrypt unregistered [ 1527.406695] Key type ._llcrypt registered [ 1527.411170] Key type .llcrypt registered [ 1529.076693] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:04:49 (1790712289) [ 1530.097489] Key type .llcrypt unregistered [ 1530.101647] Key type ._llcrypt unregistered [ 1536.947462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1537.259362] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1537.267685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1537.281105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1539.863754] Key type ._llcrypt registered [ 1539.866784] Key type .llcrypt registered [ 1540.065794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.076519] alg: No test for adler32 (adler32-zlib) [ 1541.060633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.134667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.166448] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1542.073236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1542.092814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1552.967537] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:05:13 (1790712313) [ 1553.088799] LNet: 74505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.097183] LNetError: 74505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1553.108336] LNet: Removed LNI 10.1.2.3@tcp [ 1553.119633] LNet: Removed LNI 192.168.204.43@tcp [ 1553.750419] Key type .llcrypt unregistered [ 1553.752279] Key type ._llcrypt unregistered [ 1560.610777] Key type ._llcrypt registered [ 1560.616602] Key type .llcrypt registered [ 1560.948787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1560.989467] alg: No test for adler32 (adler32-zlib) [ 1561.985499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.085841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.125840] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1563.095835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.132681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1564.390408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.43@tcp [ 1565.471305] LNetError: Unexpected error -22 connecting to 192.168.204.43@tcp at host 192.168.204.43:988 [ 1571.347434] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:05:32 (1790712332) [ 1571.476808] LNet: 75638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.486725] LNetError: 75638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1571.495286] LNet: Removed LNI 10.1.2.3@tcp [ 1571.518844] LNet: Removed LNI 192.168.204.43@tcp [ 1572.360587] Key type .llcrypt unregistered [ 1572.363922] Key type ._llcrypt unregistered [ 1580.205365] Key type ._llcrypt registered [ 1580.206905] Key type .llcrypt registered [ 1580.559359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1580.581044] alg: No test for adler32 (adler32-zlib) [ 1581.558560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.657462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1581.691892] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1582.651267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1582.692759] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 1583.947970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.43@tcp1 --nid 192.168.204.43@tcp2 [ 1590.494990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp2 [ 1590.518513] LNetError: 76479:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.43@tcp1 to 192.168.204.43@tcp2 - Route table may be misconfigured [ 1590.554798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp2 192.168.204.43@tcp1 [ 1600.032360] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:06:00 (1790712360) [ 1600.299649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.324417] LNet: 77053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.329616] LNetError: 77053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.335856] LNet: Removed LNI 192.168.204.43@tcp1 [ 1601.383549] LNet: Removed LNI 192.168.204.43@tcp2 [ 1601.449839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.586435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.624332] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1602.600856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1602.623751] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1603.531863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1603.598290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.43@tcp [ 1603.664330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp1 [ 1603.772480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp -e network_timeout [ 1603.828413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -e network_timeout [ 1603.966778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -e network_timeout [ 1604.013790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -e network_timeout [ 1604.047624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1604.069307] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1604.117365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1604.269759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1604.413542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1604.641214] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1613.449337] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:06:13 (1790712373) [ 1613.640256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.654296] LNet: 77916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.670502] LNetError: 77916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.676490] LNetError: 77916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1613.696291] LNet: Removed LNI 192.168.204.43@tcp [ 1614.771236] LNet: Removed LNI 192.168.204.43@tcp1 [ 1614.824570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.978833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1615.340438] LNet: 77953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.865070] Key type .llcrypt unregistered [ 1615.869264] Key type ._llcrypt unregistered [ 1632.790832] Key type ._llcrypt registered [ 1632.799218] Key type .llcrypt registered [ 1634.675111] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:06:34 (1790712394) [ 1635.280593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1635.322569] alg: No test for adler32 (adler32-zlib) [ 1636.389476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.370685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1641.001839] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:06:41 (1790712401) [ 1641.281558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.323562] LNet: 79050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.364819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.425192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.477147] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1642.565849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1642.612740] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1643.988688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1644.046533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp1 [ 1644.083471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp1 [ 1652.938377] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:06:53 (1790712413) [ 1655.432654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg443-server [ 1657.623635] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:06:58 (1790712418) [ 1659.682302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg443-server [ 1662.239939] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:07:02 (1790712422) [ 1664.448551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg443-server [ 1666.844335] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:07:07 (1790712427) [ 1669.092836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg443-server [ 1671.563785] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:07:11 (1790712431) [ 1673.744085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg443-server [ 1675.732988] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:07:16 (1790712436) [ 1677.457713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg443-server [ 1679.058784] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:07:19 (1790712439) [ 1681.456785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg443-server [ 1683.324826] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:07:24 (1790712444) [ 1685.442823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg443-server [ 1687.895351] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:07:28 (1790712448) [ 1689.668484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg443-server [ 1691.726691] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:07:32 (1790712452) [ 1692.005327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.043638] LNet: 82930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.047863] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.053980] LNet: Removed LNI 192.168.204.43@tcp [ 1693.115647] LNet: Removed LNI 192.168.204.43@tcp1 [ 1693.220643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.365271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.403728] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1694.393038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1694.568302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1694.829454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.846811] LNet: 83078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.857046] LNetError: 83078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.862634] LNetError: 83078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1694.884810] LNet: Removed LNI 192.168.204.43@tcp [ 1694.909957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.992688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1695.015402] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1696.097696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1696.270548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.289751] LNetError: 83223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.312301] LNet: Removed LNI 192.168.204.43@tcp [ 1696.330949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.372114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.391534] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1697.236737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1697.350386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1697.641788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.661695] LNet: 83370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.666619] LNet: 83370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1698.731172] LNet: Removed LNI 192.168.204.43@tcp [ 1698.759789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.828444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1698.870407] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1700.028524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1700.286584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.305292] LNetError: 83515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.311047] LNetError: 83515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1700.320358] LNet: Removed LNI 192.168.204.43@tcp [ 1700.346826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.426184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.475629] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1701.196958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1701.251608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1701.525051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.633299] LNet: Removed LNI 192.168.204.43@tcp [ 1702.666932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.743499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1702.768713] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1703.810462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1704.007452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.038495] LNet: 83808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.049170] LNet: 83808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1705.129212] LNet: Removed LNI 192.168.204.43@tcp [ 1705.213678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.325514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.373597] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1706.819176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1706.954490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1707.457461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.524506] LNetError: 83955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.535964] LNetError: 83955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1707.581429] LNet: Removed LNI 192.168.204.43@tcp [ 1707.618730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.762687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1707.859291] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1709.945442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1710.426619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.527687] LNet: Removed LNI 192.168.204.43@tcp [ 1711.589440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.714491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.796525] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1713.840801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1713.993282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1714.456777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.490409] LNet: 84248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.497316] LNet: 84248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1715.557994] LNet: Removed LNI 192.168.204.43@tcp [ 1715.609726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.712897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1715.758694] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1716.821478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1717.125566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.152663] LNetError: 84394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.159221] LNetError: 84394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1718.183198] LNet: Removed LNI 192.168.204.43@tcp [ 1718.225025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.317699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.359860] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1719.234710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1719.322578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1719.611513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.679749] LNet: Removed LNI 192.168.204.43@tcp [ 1720.753220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.848875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1720.889284] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1722.058397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1722.437211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.502768] LNet: Removed LNI 192.168.204.43@tcp [ 1723.554761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.644592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.687111] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1724.927498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1725.003485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.048235] LNet: Removed LNI 192.168.204.43@tcp [ 1725.073052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.131915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.162876] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1726.140269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1726.353132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.409767] LNet: Removed LNI 192.168.204.43@tcp [ 1726.437419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.504875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1726.593207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.646055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.726093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1726.869599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.941909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.014728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1727.138291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.198926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.264722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1727.294290] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1728.389512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.438460] LNet: Removed LNI 192.168.204.43@tcp [ 1728.492810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.679768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1736.448838] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:08:17 (1790712497) [ 1736.825572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.879598] LNet: 85709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.889896] LNet: 85709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1736.923770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.014771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1737.055460] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 1738.679878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.735835] LNetError: 85847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.745755] LNetError: 85847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1738.773817] LNet: Removed LNI 192.168.204.43@tcp231 [ 1738.817357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.907766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1738.938970] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 1738.945730] LNet: 85855:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1739.512071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1739.555965] LNet: Removed LNI 192.168.204.43@tcp231 [ 1739.579454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1739.616729] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 1740.860546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.908178] LNet: Removed LNI 192.168.204.43@tcp231 [ 1740.953798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.053093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1741.092523] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 1741.102216] LNet: 86007:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1741.726914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1741.767865] LNet: Removed LNI 192.168.204.43@tcp231 [ 1741.784497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1741.821355] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 1743.181592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.233193] LNet: Removed LNI 192.168.204.43@tcp231 [ 1743.267097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.369444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1743.406679] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 1743.412683] LNet: 86159:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1743.996165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.039857] LNet: Removed LNI 192.168.204.43@tcp231 [ 1744.064503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1744.096801] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 1745.598337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.629495] LNet: Removed LNI 192.168.204.43@tcp231 [ 1745.647542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.691559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1745.712056] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 1745.717862] LNet: 86310:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.198401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1746.264242] LNet: Removed LNI 192.168.204.43@tcp231 [ 1746.299313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1746.385630] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 1758.717412] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:08:39 (1790712519) [ 1758.805024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.892118] LNet: Removed LNI 192.168.204.43@tcp231 [ 1758.940702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.112811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1759.152826] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 1760.560414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1760.625466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1760.701441] LNet: Removed LNI 192.168.204.43@tcp232 [ 1760.764381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1760.799680] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 1762.255716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1762.322377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1762.378684] LNet: Removed LNI 192.168.204.43@tcp232 [ 1762.409085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1762.448897] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 1763.455264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1763.495392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1763.527938] LNet: Removed LNI 192.168.204.43@tcp232 [ 1763.548566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1763.585755] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 1764.382807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1764.456352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1764.492919] LNet: Removed LNI 192.168.204.43@tcp232 [ 1764.548392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1772.881532] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:08:53 (1790712533) [ 1773.708783] LNet: 88150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.715607] LNet: 88150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1774.666934] Key type .llcrypt unregistered [ 1774.674775] Key type ._llcrypt unregistered [ 1783.399437] Key type ._llcrypt registered [ 1783.406874] Key type .llcrypt registered [ 1783.812917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.839882] alg: No test for adler32 (adler32-zlib) [ 1784.856982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1784.909490] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1794.326368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1803.938104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.016216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1804.051931] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1805.382223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1806.533422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 1806.599313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 1806.640780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 1806.714294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 1806.798810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 1806.852908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 1806.920308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 1806.997525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 1807.042419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1807.075608] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1807.080391] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1807.391284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.445300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.491885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.558262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.608937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1807.677864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.766963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.861177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.943654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.993178] LNet: 88819:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.024667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1808.089890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1808.155566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1808.226337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1808.274647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1808.315920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.889070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 1808.924805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 1808.957495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 1808.990826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 1809.030962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 1809.102328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 1809.162739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 1809.207695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 1809.247968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1809.289843] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1809.577364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1809.628732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1809.684690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1809.714210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1809.764716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1809.796735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1809.831319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1809.866283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1809.890895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1809.914730] LNet: 88941:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1809.935748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1809.969588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1809.997417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.029696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.075328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1810.151722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1810.819486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 1810.873593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 1810.921897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 1811.003363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 1811.076259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 1811.142837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 1811.213298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 1811.289277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 1811.351701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1811.372988] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1811.737503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1811.778807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1811.809695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.852529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.891370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1811.939628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.992300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.048929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.089903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.110768] LNet: 89064:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.142198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.174813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.239466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.283425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.349236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.415544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.140806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 1813.212649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 1813.284229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 1813.355751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 1813.390684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 1813.429942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 1813.514048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 1813.594746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 1813.635806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1813.661723] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1814.064769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1814.107678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1814.145800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1814.182801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1814.250884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1814.290963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1814.327842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1814.360151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1814.387118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1814.405885] LNet: 89186:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.431831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.461777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.542368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.588525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1814.650786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1814.714248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1815.345676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 1815.394702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 1815.429904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 1815.472226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 1815.543788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 1815.614667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 1815.676373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 1815.717755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 1815.765456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1816.117701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1816.157554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1816.228603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1816.279296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1816.348825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1816.410693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1816.452414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1816.482212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1816.519358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1816.594972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1816.648190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1816.698755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1816.760261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1816.819948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1816.863444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1817.525826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 1817.562768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 1817.593296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 1817.629416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 1817.688756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 1817.727260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 1817.765906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 1817.818129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 1817.865436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 1817.890730] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1817.898878] LNet: Skipped 1 previous similar message [ 1817.912433] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 1818.151360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1818.180705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1818.209607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1818.235491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1818.261514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1818.290184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1818.321588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1818.353092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1818.383698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1818.413241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1818.437787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1818.468953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1818.500123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1818.533657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1818.567316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1825.178653] LNet: 89505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.189200] LNetError: 89505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.200384] LNet: Removed LNI 192.168.204.43@tcp [ 1825.233279] LNet: Removed LNI 192.168.204.43@tcp1 [ 1825.740777] Key type .llcrypt unregistered [ 1825.748989] Key type ._llcrypt unregistered [ 1844.915332] Key type ._llcrypt registered [ 1844.917687] Key type .llcrypt registered [ 1846.262889] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:10:06 (1790712606) [ 1846.665598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.698794] alg: No test for adler32 (adler32-zlib) [ 1847.576360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.625583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.648852] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1848.499481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1848.526824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 1848.547522] LNetError: 90484:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1848.610542] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1848.638811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.43@tcp [ 1848.763345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1854.935993] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:10:15 (1790712615) [ 1855.201543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.227077] LNet: 91073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.245620] LNetError: 91073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.294273] LNet: Removed LNI 192.168.204.43@tcp [ 1856.328643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.429669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.467870] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1857.255341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1857.269860] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1858.350624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.43@tcp [ 1858.382757] LNetError: 91340:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1858.417927] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1858.571820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.43@tcp [ 1867.239788] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:10:28 (1790712628) [ 1869.679432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg443-server [ 1871.739884] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:10:32 (1790712632) [ 1874.179572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg443-server [ 1876.128280] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:10:36 (1790712636) [ 1877.852213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg443-server [ 1879.692332] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:10:40 (1790712640) [ 1880.270245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.295345] LNet: 92962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.308917] LNetError: 92962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.317989] LNet: Removed LNI 192.168.204.43@tcp [ 1880.329586] LNet: Removed LNI 192.168.204.43@tcp1 [ 1880.364876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.418724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.447969] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1881.125509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1881.203623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1881.241277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1881.341511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1881.390338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1881.461432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1889.937830] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:10:50 (1790712650) [ 1890.058112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.105861] LNet: 93689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.120730] LNetError: 93689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.136295] LNetError: 93689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1890.162119] LNet: Removed LNI 192.168.204.43@tcp [ 1890.229985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.308711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1890.348736] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1897.542669] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:10:58 (1790712658) [ 1897.787162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.804453] LNet: 94397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.810121] LNetError: 94397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.828695] LNet: Removed LNI 192.168.204.43@tcp [ 1897.854945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.919934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1897.948581] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1904.849829] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:11:05 (1790712665) [ 1906.409328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1908.791922] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:11:09 (1790712669) [ 1909.383362] LNet: 95432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.394493] LNetError: 95432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.408732] LNet: Removed LNI 192.168.204.43@tcp [ 1910.191916] Key type .llcrypt unregistered [ 1910.195636] Key type ._llcrypt unregistered [ 1917.027633] Key type ._llcrypt registered [ 1917.030346] Key type .llcrypt registered [ 1917.191598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1917.210680] alg: No test for adler32 (adler32-zlib) [ 1918.174047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1918.221547] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1926.978757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1933.258944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.143@tcp [ 1934.743205] LNet: 95850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.749313] LNetError: 95850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.760774] LNet: Removed LNI 192.168.204.43@tcp [ 1935.144608] Key type .llcrypt unregistered [ 1935.146621] Key type ._llcrypt unregistered [ 1947.997599] Key type ._llcrypt registered [ 1948.003381] Key type .llcrypt registered [ 1949.330369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1951.755645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1954.025545] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:11:54 (1790712714) [ 1955.584903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1957.503450] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:11:58 (1790712718) [ 1959.373258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg443-server [ 1961.157665] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:12:02 (1790712722) [ 1963.171260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg443-server [ 1965.155767] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:12:06 (1790712726) [ 1967.238179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.347727] alg: No test for adler32 (adler32-zlib) [ 1968.402865] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1968.648233] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1970.343285] Key type lgssc registered [ 1972.070856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1995.502868] Key type lgssc unregistered [ 1995.755714] LNet: 99033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.765918] LNetError: 99033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.783182] LNet: Removed LNI 192.168.204.43@tcp [ 1996.402632] Key type .llcrypt unregistered [ 1996.405730] Key type ._llcrypt unregistered [ 2011.042644] Key type ._llcrypt registered [ 2011.045353] Key type .llcrypt registered [ 2012.506796] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:12:53 (1790712773) [ 2012.968255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.992698] alg: No test for adler32 (adler32-zlib) [ 2013.917598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.980563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2014.019493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2014.056359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2014.094366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2021.376886] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:13:01 (1790712781) [ 2021.530307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.593854] LNet: 100450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.637519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.720955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.745226] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2023.192682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.228200] LNet: 100600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.234267] LNetError: 100600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.255175] LNet: Removed LNI 192.168.204.43@tcp [ 2023.279459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.345499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2023.386306] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2024.080597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.130784] LNetError: 100628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.151728] LNet: Removed LNI 192.168.204.43@tcp [ 2024.194359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.309958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2024.348270] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2031.734654] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:13:12 (1790712792) [ 2032.036845] LNet: 101219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.045424] LNet: 101219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2032.056966] LNetError: 101219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.079643] LNet: Removed LNI 192.168.204.43@tcp [ 2032.786384] Key type .llcrypt unregistered [ 2032.788843] Key type ._llcrypt unregistered [ 2040.136310] Key type ._llcrypt registered [ 2040.141264] Key type .llcrypt registered [ 2040.529651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.565166] alg: No test for adler32 (adler32-zlib) [ 2041.490924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.550856] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2041.553832] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.260933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2042.276846] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.279443] LNet: Skipped 2 previous similar messages [ 2042.285158] Ping reply received with mapping: [ 2042.286961] found metadata reply NI[192.168.204.43@tcp] [ 2042.289480] 2b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.292653] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.299989] 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.443745] LNet: 101616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.457816] LNetError: 101616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.476505] LNet: Removed LNI 192.168.204.43@tcp [ 2043.193466] Key type .llcrypt unregistered [ 2043.199677] Key type ._llcrypt unregistered [ 2055.510836] Key type ._llcrypt registered [ 2055.513617] Key type .llcrypt registered [ 2056.880416] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:13:37 (1790712817) [ 2057.471822] Key type .llcrypt unregistered [ 2057.479152] Key type ._llcrypt unregistered [ 2065.885532] Key type ._llcrypt registered [ 2065.887712] Key type .llcrypt registered [ 2066.107864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2066.132778] alg: No test for adler32 (adler32-zlib) [ 2067.421484] LNetError: 102657:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2068.019215] Key type .llcrypt unregistered [ 2068.021970] Key type ._llcrypt unregistered [ 2073.712206] Key type ._llcrypt registered [ 2073.714779] Key type .llcrypt registered [ 2075.153477] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:13:55 (1790712835) [ 2075.504299] Key type .llcrypt unregistered [ 2075.506483] Key type ._llcrypt unregistered [ 2081.351111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2081.354453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.973210] Key type ._llcrypt registered [ 2083.981815] Key type .llcrypt registered [ 2084.235143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.270140] alg: No test for adler32 (adler32-zlib) [ 2085.224915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.295948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2096.083818] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:14:16 (1790712856) [ 2096.232858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.276039] LNet: 104433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.281693] LNetError: 104433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2096.319702] LNet: Removed LNI 10.1.2.3@tcp [ 2096.377296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.563582] Key type .llcrypt unregistered [ 2097.566751] Key type ._llcrypt unregistered [ 2104.980856] Key type ._llcrypt registered [ 2104.983018] Key type .llcrypt registered [ 2105.172481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2105.194117] alg: No test for adler32 (adler32-zlib) [ 2106.102449] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2106.150438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2106.349128] LNet: 104726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.356975] LNetError: 104726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.377870] LNet: Removed LNI 192.168.204.43@tcp [ 2106.973767] Key type .llcrypt unregistered [ 2106.978481] Key type ._llcrypt unregistered [ 2118.402534] Key type ._llcrypt registered [ 2118.404268] Key type .llcrypt registered [ 2119.530196] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:14:40 (1790712880) [ 2120.381910] Key type .llcrypt unregistered [ 2120.385504] Key type ._llcrypt unregistered [ 2126.879484] Key type ._llcrypt registered [ 2126.882661] Key type .llcrypt registered [ 2127.184432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2127.220850] alg: No test for adler32 (adler32-zlib) [ 2134.277753] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:14:55 (1790712895) [ 2134.542586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.573972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.621389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.640343] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2135.199269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.226669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2135.272759] LNet: Removed LNI 192.168.204.43@tcp [ 2135.489745] LNet: 106469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.822649] Key type .llcrypt unregistered [ 2135.827418] Key type ._llcrypt unregistered [ 2147.463320] Key type ._llcrypt registered [ 2147.466685] Key type .llcrypt registered [ 2148.752961] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:15:09 (1790712909) [ 2149.367283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.398021] alg: No test for adler32 (adler32-zlib) [ 2150.329595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.399848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.422172] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2151.231948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2151.261810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.43@tcp [ 2151.280790] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2151.284177] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2151.287755] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2151, 0, 0 [ 2151.297215] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2151.299839] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2151.303958] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2151.306965] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2151, 0, 0 [ 2151.311979] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2151.316501] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2151.326822] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2151.330332] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2151, 0, 2201 [ 2151.335255] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2151.337789] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2151.341480] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2151.347475] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2151, 0, 2201 [ 2151.352756] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2158.290911] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:15:19 (1790712919) [ 2158.814916] LNet: 108009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.822458] LNetError: 108009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.845764] LNet: Removed LNI 192.168.204.43@tcp [ 2159.577510] Key type .llcrypt unregistered [ 2159.580895] Key type ._llcrypt unregistered [ 2166.925527] Key type ._llcrypt registered [ 2166.928375] Key type .llcrypt registered [ 2167.116647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.141338] alg: No test for adler32 (adler32-zlib) [ 2168.073646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2168.116315] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2176.275988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2186.768297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.848112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2186.878531] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 2187.602167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2187.840108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2188.054506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2188.057628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2188.065554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2190.397303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2190.423989] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2201.151488] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:16:02 (1790712962) [ 2201.456521] LNet: 109525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.463283] LNetError: 109525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.471271] LNet: Removed LNI 192.168.204.43@tcp [ 2202.527637] LNet: Removed LNI 192.168.204.43@tcp1 [ 2202.544670] LNetError: 109525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2202.558327] LNetError: 109525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2202.599601] LNet: Removed LNI 10.1.2.3@tcp99 [ 2203.359587] Key type .llcrypt unregistered [ 2203.368106] Key type ._llcrypt unregistered [ 2210.278780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.283862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2212.880892] Key type ._llcrypt registered [ 2212.883886] Key type .llcrypt registered [ 2213.086203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2213.105106] alg: No test for adler32 (adler32-zlib) [ 2214.041929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.112788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.132822] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2214.742873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2214.761879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2216.656401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2216.708333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2216.751695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2216.780357] LNetError: 110268:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2216.812973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2216.888865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2217.317942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2217.377892] LNet: 110289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.383769] LNetError: 110289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2217.390275] LNet: Removed LNI 10.1.2.3@tcp [ 2218.416871] LNet: Removed LNI 192.168.204.43@tcp [ 2219.166739] Key type .llcrypt unregistered [ 2219.170178] Key type ._llcrypt unregistered [ 2231.445603] Key type ._llcrypt registered [ 2231.448866] Key type .llcrypt registered [ 2232.740812] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:16:33 (1790712993) [ 2233.087378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.100267] alg: No test for adler32 (adler32-zlib) [ 2234.068240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.152598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.196673] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2234.895749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2235.004446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg443-client@tcp [ 2241.837939] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:16:42 (1790713002) [ 2241.935920] LNet: 111814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.946885] LNetError: 111814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.971769] LNet: Removed LNI 192.168.204.43@tcp [ 2242.589644] Key type .llcrypt unregistered [ 2242.594194] Key type ._llcrypt unregistered [ 2248.910211] Key type ._llcrypt registered [ 2248.912601] Key type .llcrypt registered [ 2249.125083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.142160] alg: No test for adler32 (adler32-zlib) [ 2250.027136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2250.072680] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2250.104871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2256.128971] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:16:57 (1790713017) [ 2257.477243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2259.313291] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:17:00 (1790713020) [ 2260.829314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2263.357084] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:17:03 (1790713023) [ 2263.476515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.500408] LNet: 113238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.505957] LNetError: 113238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.557345] LNet: Removed LNI 192.168.204.43@tcp [ 2263.591108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.758246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.800898] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2265.066416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2265.199962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2272.883352] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:17:13 (1790713033) [ 2273.080092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.125146] LNet: 113952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.136111] LNetError: 113952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.155293] LNet: Removed LNI 192.168.204.43@tcp [ 2273.178477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.252374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2273.294575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2273.834335] Key type .llcrypt unregistered [ 2273.838151] Key type ._llcrypt unregistered [ 2285.408936] Key type ._llcrypt registered [ 2285.415556] Key type .llcrypt registered [ 2286.537720] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:17:27 (1790713047) [ 2287.318889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.352360] alg: No test for adler32 (adler32-zlib) [ 2288.258097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.349759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.380339] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2289.047384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2289.101583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2296.079377] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:17:36 (1790713056) [ 2296.211379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.238357] LNet: 115501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.253048] LNetError: 115501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.266057] LNet: Removed LNI 192.168.204.43@tcp [ 2296.316696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.377966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2296.408677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2303.364657] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:17:44 (1790713064) [ 2303.575233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.617920] LNet: 116082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.656959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.787866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2304.204273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2304.242587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2304.331318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2304.730696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2304.767887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2304.856817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.885580] LNet: 116120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.917467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.999631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2305.389671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2305.415431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2305.461733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2305.988505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2306.015145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2313.349040] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:17:53 (1790713073) [ 2313.665586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.685611] LNet: 116720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.726782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.814175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.865403] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2314.759620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2314.792213] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2315.722815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2315.801368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2316.024672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.054254] LNetError: 117000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.059952] LNet: Removed LNI 192.168.204.43@tcp [ 2316.070861] LNet: Removed LNI 192.168.204.43@tcp2 [ 2316.107647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.199761] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2316.210170] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2316.219522] LNet: 117006:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2316.706520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2316.935495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.983174] LNet: Removed LNI 192.168.204.43@tcp [ 2317.988739] LNetError: 117043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.998833] LNetError: 117043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2318.056116] LNet: Removed LNI 192.168.204.43@tcp2 [ 2318.102410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.167991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2318.197680] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2319.103495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2319.137628] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2320.166412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2320.225242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2320.332753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2320.736786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.772834] LNet: 117327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.778739] LNet: 117327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2320.793963] LNetError: 117327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.805992] LNet: Removed LNI 192.168.204.43@tcp [ 2321.835070] LNet: Removed LNI 192.168.204.43@tcp2 [ 2321.918827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.013664] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2322.021164] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2322.038287] LNet: 117333:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2322.750769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2322.992202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.027527] LNet: Removed LNI 192.168.204.43@tcp [ 2324.070220] LNet: Removed LNI 192.168.204.43@tcp2 [ 2324.124360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.193623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.235597] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2324.989774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2325.019216] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2325.799635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2325.853588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2326.054255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.072825] LNetError: 117650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.077817] LNetError: 117650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2326.083931] LNet: Removed LNI 192.168.204.43@tcp [ 2326.103307] LNet: Removed LNI 192.168.204.43@tcp2 [ 2326.121922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.174807] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2326.181320] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2326.185955] LNet: 117656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2326.744453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2326.964593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.033294] LNet: Removed LNI 192.168.204.43@tcp [ 2328.064192] LNet: Removed LNI 192.168.204.43@tcp2 [ 2328.111913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.185195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.219900] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2329.033908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2329.052680] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2329.957732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2329.992350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2330.062823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2330.303223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.322215] LNet: 117978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.325766] LNet: 117978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2331.359177] LNet: Removed LNI 192.168.204.43@tcp [ 2332.389954] LNet: Removed LNI 192.168.204.43@tcp2 [ 2332.419397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.499222] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2332.507392] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2332.514423] LNet: 117984:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.085146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2333.252609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.286965] LNet: Removed LNI 192.168.204.43@tcp [ 2334.313726] LNet: Removed LNI 192.168.204.43@tcp2 [ 2334.366726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.442682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.482856] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2335.120781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2335.148479] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2336.104590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2336.165516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2336.217322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2336.919431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.955099] LNetError: 118318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.962919] LNetError: 118318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2336.971858] LNet: Removed LNI 192.168.204.43@tcp [ 2336.986507] LNet: Removed LNI 192.168.204.43@tcp2 [ 2337.059021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.158951] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2337.170151] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2337.196552] LNet: 118324:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.043417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2338.428026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.487178] LNet: Removed LNI 192.168.204.43@tcp [ 2339.506276] LNet: Removed LNI 192.168.204.43@tcp2 [ 2339.550292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.647727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2339.673132] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2340.377460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2340.402800] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2341.146527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2341.179536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2341.201912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2341.246086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2341.583247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.623164] LNet: Removed LNI 192.168.204.43@tcp [ 2343.654600] LNet: Removed LNI 192.168.204.43@tcp2 [ 2343.705158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.771081] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2343.776439] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2344.295784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2350.459721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2352.141167] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:18:33 (1790713113) [ 2352.319986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.348355] LNet: 119440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.352620] LNet: 119440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2352.357862] LNet: Removed LNI 192.168.204.43@tcp [ 2353.387433] LNet: Removed LNI 192.168.204.43@tcp2 [ 2353.421255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.815892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.865510] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2354.843578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2354.863995] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2356.140432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.184135] LNetError: 119716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2356.202875] LNetError: 119716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2356.221860] LNet: Removed LNI 10.1.2.3@tcp [ 2356.241106] LNet: Removed LNI 192.168.204.43@tcp [ 2356.272516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.351970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2356.386738] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2356.396746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2356.970543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2356.992618] LNet: Removed LNI 192.168.204.43@tcp [ 2357.006949] LNet: Removed LNI 10.1.2.3@tcp [ 2364.544349] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:18:45 (1790713125) [ 2364.676494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.745902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.867987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.910084] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2365.787549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2365.832852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2365.896539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2365.927064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2365.994630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.032943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.064776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.112603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.162533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.206529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.247846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.276896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2366.429769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2373.091253] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 16:18:53 (1790713133) [ 2373.267397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.310766] LNet: Removed LNI 192.168.204.43@tcp [ 2374.363916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.449373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2374.479477] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2375.891954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2375.986599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2376.076552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.150665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.225821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.291299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.421939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.513589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.560955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.608707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.651925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.731197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2376.906064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2376.959436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.029496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.085611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.137682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.255411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.422820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.526752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.617481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.745493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2377.822662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2386.730524] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:19:06 (1790713146) [ 2386.824380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.847483] LNet: 121807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.860486] LNet: 121807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2387.952320] LNet: Removed LNI 192.168.204.43@tcp [ 2388.007731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.155530] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2388.332116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2388.334785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2388.337526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2390.708540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.744486] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2391.477899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2391.494619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2393.685420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2393.744218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2393.839422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.43@tcp -m PUT -e local_timeout -r 1 [ 2393.896393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2393.963409] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2393.965862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2394.267665] LNetError: 121881:0:(lib-move.c:4794:lnet_parse()) 192.168.204.43@tcp, src 192.168.204.43@tcp: Dropping PUT (error -108 looking up sender) [ 2395.305872] LNetError: 122340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2395.316130] LNetError: 122340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2395.336935] LNet: Removed LNI 10.1.2.3@tcp [ 2396.388838] LNet: Removed LNI 192.168.204.43@tcp [ 2396.901458] Key type .llcrypt unregistered [ 2396.904351] Key type ._llcrypt unregistered [ 2410.824588] Key type ._llcrypt registered [ 2410.828895] Key type .llcrypt registered [ 2412.115290] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:19:32 (1790713172) [ 2412.710461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.731190] alg: No test for adler32 (adler32-zlib) [ 2413.821966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.419993] LNet: 123193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.236265] Key type .llcrypt unregistered [ 2415.238089] Key type ._llcrypt unregistered [ 2422.485901] Key type ._llcrypt registered [ 2422.490240] Key type .llcrypt registered [ 2422.640940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2422.652784] alg: No test for adler32 (adler32-zlib) [ 2423.689742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2423.744929] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2432.177324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2432.449774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2433.327289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 2433.578262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.624265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.681601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.719812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.766704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2433.806797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.846375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.903325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.960041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.988349] LNet: 123645:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.020218] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 2434.030174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.072425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.109923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.159881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.201160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.255405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.008827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2435.285800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2435.320658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2435.355721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.393729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.429573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2435.462299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.492971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.524179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.559767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.579148] LNet: 123764:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.603462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.640561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.673478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.727768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.768593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.835702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.602592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 2436.950301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.985541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2437.047331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2437.084582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2437.132288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2437.171265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2437.232744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2437.285724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2437.322250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2437.335870] LNet: 123884:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2437.351290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2437.402289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2437.439213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2437.485047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2437.511644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.535039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2438.156828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2438.480376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2438.529381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2438.561571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2438.582970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2438.608863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2438.631165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2438.654701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2438.706344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2438.788774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2438.848824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2438.907962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2438.966278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2439.035928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2439.085238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2439.130702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2439.464330] LNet: 124049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.471401] LNetError: 124049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.488293] LNet: Removed LNI 192.168.204.43@tcp [ 2440.006038] Key type .llcrypt unregistered [ 2440.007816] Key type ._llcrypt unregistered [ 2460.857664] Key type ._llcrypt registered [ 2460.859550] Key type .llcrypt registered [ 2462.142872] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:20:22 (1790713222) [ 2462.848667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.889337] alg: No test for adler32 (adler32-zlib) [ 2463.931178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.547985] LNet: 124937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2465.435437] Key type .llcrypt unregistered [ 2465.437631] Key type ._llcrypt unregistered [ 2473.857333] Key type ._llcrypt registered [ 2473.859369] Key type .llcrypt registered [ 2474.185689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2474.205838] alg: No test for adler32 (adler32-zlib) [ 2475.265752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2475.328254] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2484.938529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2495.546431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.585944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2495.619638] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 2496.836935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2497.027979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2497.771816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 2497.887115] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 2498.038279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2498.073874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2498.104889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2498.145838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.234227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2498.292865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.333079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.376489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.430482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.452758] LNet: 125597:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.489750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.553310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.604915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.669894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.725473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.765092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2499.684413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2500.007979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2500.058912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2500.104840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2500.162585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2500.207721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2500.251392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2500.326691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2500.388822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2500.429746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2500.456684] LNet: 125717:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2500.503558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2500.559688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2500.657124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2500.746949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2500.823970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2500.905733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2501.960553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 2502.407180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2502.468187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2502.512585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2502.583738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2502.640903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2502.688864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2502.731437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2502.782162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2502.813295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2502.835346] LNet: 125836:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.872786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2502.951320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2503.046500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2503.152111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2503.202413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2503.274363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2504.209579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2504.539892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2504.572317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2504.614323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2504.659156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2504.731161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2504.779874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2504.842786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2504.903057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2504.954516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2504.984725] LNet: 125956:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2505.019393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2505.074814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2505.115651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2505.178169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2505.260693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2505.330978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2511.561714] LNet: 126037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.570714] LNetError: 126037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.578383] LNet: Removed LNI 192.168.204.43@tcp [ 2511.588864] LNet: Removed LNI 192.168.204.43@tcp1 [ 2512.102477] Key type .llcrypt unregistered [ 2512.106889] Key type ._llcrypt unregistered [ 2531.237443] Key type ._llcrypt registered [ 2531.240662] Key type .llcrypt registered [ 2533.183020] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:21:33 (1790713293) [ 2533.672088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.693789] alg: No test for adler32 (adler32-zlib) [ 2534.679438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2534.752665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2534.789527] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2535.925385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2535.978485] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2537.021570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2537.112881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2538.143289] LNet: Removed LNI 192.168.204.43@tcp [ 2546.280661] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:21:46 (1790713306) [ 2548.133715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2550.012492] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:21:50 (1790713310) [ 2558.062928] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:21:58 (1790713318) [ 2566.730810] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:22:07 (1790713327) [ 2575.629655] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:22:16 (1790713336) [ 2582.516138] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:22:23 (1790713343) [ 2590.415553] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:22:31 (1790713351) [ 2599.188507] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:22:39 (1790713359) [ 2606.350945] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:22:47 (1790713367) [ 2614.364376] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:22:54 (1790713374) [ 2621.810355] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:23:02 (1790713382) [ 2628.635602] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:23:09 (1790713389) [ 2636.875924] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:23:17 (1790713397) [ 2644.888244] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:23:25 (1790713405) [ 2653.444359] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:23:33 (1790713413) [ 2661.285564] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:23:41 (1790713421) [ 2670.420560] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:23:50 (1790713430) [ 2677.694611] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:23:58 (1790713438) [ 2685.273164] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:24:06 (1790713446) [ 2685.579345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2685.613464] LNet: 137734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2685.622497] LNetError: 137734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.641410] LNet: Removed LNI 192.168.204.43@tcp2 [ 2685.674387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2685.802609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2685.835047] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2687.199394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2687.228797] LNet: 137876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.238363] LNetError: 137876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.267738] LNet: Removed LNI 192.168.204.43@tcp [ 2687.298533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2687.368510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2687.393519] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2694.962612] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:24:16 (1790713456) [ 2704.483464] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:24:24 (1790713464) [ 2713.235344] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:24:33 (1790713473) [ 2723.036307] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:24:43 (1790713483) [ 2731.589649] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:24:52 (1790713492) [ 2740.445559] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:25:01 (1790713501) [ 2747.249799] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:25:07 (1790713507) [ 2754.627099] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:25:15 (1790713515) [ 2761.436987] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:25:22 (1790713522) [ 2767.990794] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:25:29 (1790713529) [ 2774.109354] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:25:34 (1790713534) [ 2780.923948] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:25:41 (1790713541) [ 2787.133949] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2549 sec ========== 16:25:48 (1790713548) [ 2787.462779] LNet: 145342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.468591] LNetError: 145342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.486173] LNet: Removed LNI 192.168.204.43@tcp [ 2788.078179] Key type .llcrypt unregistered [ 2788.081475] Key type ._llcrypt unregistered [ 2799.584628] Key type ._llcrypt registered [ 2799.594238] Key type .llcrypt registered [ 2800.006846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2800.023159] alg: No test for adler32 (adler32-zlib) [ 2801.083747] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2801.289457] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2802.992720] Key type lgssc registered [ 2804.548461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2868.917591] Lustre: Mounted lustre-client - version 2.17.59_2_g598166e [ 2873.844779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2880.943454] Lustre: Unmounted lustre-client [ 2911.207915] Key type lgssc unregistered [ 2911.545940] LNet: 148300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.554146] LNetError: 148300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2911.578031] LNet: Removed LNI 192.168.204.43@tcp [ 2912.326309] Key type .llcrypt unregistered [ 2912.328114] Key type ._llcrypt unregistered