[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473602500 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003174] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013089] *** VALIDATE selinux *** [ 0.021761] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026206] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030134] *** VALIDATE tmpfs *** [ 0.031475] *** VALIDATE proc *** [ 0.033174] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037073] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042843] debug: unmapping init [mem 0xffffffffb6a59000-0xffffffffb6a60fff] [ 0.044764] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053307] rcu: Hierarchical SRCU implementation. [ 0.055445] smp: Bringing up secondary CPUs ... [ 0.056497] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060151] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.166435] node 0 deferred pages initialised in 102ms [ 0.170157] devtmpfs: initialized [ 0.171316] x86/mm: Memory block size: 128MB [ 0.175170] gcov: version magic: 0x41383552 [ 0.178291] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.181128] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183334] pinctrl core: initialized pinctrl subsystem [ 0.185164] [ 0.185648] ************************************************************* [ 0.187012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188010] ** ** [ 0.190009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.191011] ** ** [ 0.192013] ** This means that this kernel is built to expose internal ** [ 0.194009] ** IOMMU data structures, which may compromise security on ** [ 0.195006] ** your system. ** [ 0.196009] ** ** [ 0.198012] ** If you see this message and you are not debugging the ** [ 0.199009] ** kernel, report this immediately to your vendor! ** [ 0.201019] ** ** [ 0.202008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203010] ************************************************************* [ 0.204603] NET: Registered protocol family 16 [ 0.206405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213120] cpuidle: using governor menu [ 0.214794] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218782] PCI: Using configuration type 1 for base access [ 0.220163] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235104] cryptd: max_cpu_qlen set to 1000 [ 0.238133] ACPI: Added _OSI(Module Device) [ 0.239011] ACPI: Added _OSI(Processor Device) [ 0.240010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240916] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243886] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248362] ACPI: Interpreter enabled [ 0.249047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249808] ACPI: Using IOAPIC for interrupt routing [ 0.251080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252353] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260033] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268546] acpiphp: Slot [2] registered [ 0.269107] acpiphp: Slot [5] registered [ 0.270101] acpiphp: Slot [6] registered [ 0.270984] acpiphp: Slot [3] registered [ 0.272079] acpiphp: Slot [4] registered [ 0.273135] acpiphp: Slot [7] registered [ 0.273945] acpiphp: Slot [8] registered [ 0.275081] acpiphp: Slot [9] registered [ 0.275877] acpiphp: Slot [10] registered [ 0.277060] acpiphp: Slot [11] registered [ 0.277956] acpiphp: Slot [12] registered [ 0.279073] acpiphp: Slot [13] registered [ 0.279828] acpiphp: Slot [14] registered [ 0.280108] acpiphp: Slot [15] registered [ 0.281014] acpiphp: Slot [16] registered [ 0.281975] acpiphp: Slot [17] registered [ 0.283105] acpiphp: Slot [18] registered [ 0.283905] acpiphp: Slot [19] registered [ 0.284054] acpiphp: Slot [20] registered [ 0.284906] acpiphp: Slot [21] registered [ 0.286068] acpiphp: Slot [22] registered [ 0.286844] acpiphp: Slot [23] registered [ 0.288102] acpiphp: Slot [24] registered [ 0.288979] acpiphp: Slot [25] registered [ 0.290076] acpiphp: Slot [26] registered [ 0.290940] acpiphp: Slot [27] registered [ 0.292064] acpiphp: Slot [28] registered [ 0.292786] acpiphp: Slot [29] registered [ 0.293065] acpiphp: Slot [30] registered [ 0.293951] acpiphp: Slot [31] registered [ 0.295049] PCI host bridge to bus 0000:00 [ 0.295822] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.303019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305746] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308010] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312846] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.315551] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324262] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326871] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332863] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.349017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.358458] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.391000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.395017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.400014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.413016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421702] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.424384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.426445] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.429265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.431225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.435154] iommu: Default domain type: Passthrough [ 0.436567] SCSI subsystem initialized [ 0.437224] ACPI: bus type USB registered [ 0.439144] usbcore: registered new interface driver usbfs [ 0.440095] usbcore: registered new interface driver hub [ 0.442126] usbcore: registered new device driver usb [ 0.444255] pps_core: LinuxPPS API ver. 1 registered [ 0.446019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.450107] PTP clock support registered [ 0.452110] EDAC MC: Ver: 3.0.0 [ 0.454313] PCI: Using ACPI for IRQ routing [ 0.456132] NetLabel: Initializing [ 0.457020] NetLabel: domain hash size = 128 [ 0.459017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.461119] NetLabel: unlabeled traffic allowed by default [ 0.463364] vgaarb: loaded [ 0.465300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.467020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.473072] clocksource: Switched to clocksource kvm-clock [ 0.575747] VFS: Disk quotas dquot_6.6.0 [ 0.577453] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.580142] *** VALIDATE ramfs *** [ 0.581384] *** VALIDATE hugetlbfs *** [ 0.582974] pnp: PnP ACPI init [ 0.585366] pnp: PnP ACPI: found 6 devices [ 0.602405] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.605695] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.607969] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.609911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.612288] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.614506] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.617303] NET: Registered protocol family 2 [ 0.619678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.624386] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.627699] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.632672] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.635993] TCP: Hash tables configured (established 65536 bind 65536) [ 0.638961] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.641599] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.644169] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.646670] NET: Registered protocol family 1 [ 0.649185] RPC: Registered named UNIX socket transport module. [ 0.651319] RPC: Registered udp transport module. [ 0.653027] RPC: Registered tcp transport module. [ 0.654452] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.656477] NET: Registered protocol family 44 [ 0.658016] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.660058] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.661904] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.663820] PCI: CLS 0 bytes, default 64 [ 0.665232] Unpacking initramfs... [ 2.050820] debug: unmapping init [mem 0xffff8a50bcc64000-0xffff8a50bffcffff] [ 2.054708] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.057010] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.060081] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.519698] Initialise system trusted keyrings [ 2.520574] Key type blacklist registered [ 2.521721] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.527436] zbud: loaded [ 2.529598] *** VALIDATE nfs *** [ 2.530267] *** VALIDATE nfs4 *** [ 2.531197] pstore: using deflate compression [ 2.533464] Platform Keyring initialized [ 2.618298] NET: Registered protocol family 38 [ 2.619612] Key type asymmetric registered [ 2.620765] Asymmetric key parser 'x509' registered [ 2.621985] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.626283] io scheduler mq-deadline registered [ 2.627465] io scheduler kyber registered [ 2.628698] io scheduler bfq registered [ 2.629835] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.631982] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.633878] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.635829] ACPI: Power Button [PWRF] [ 2.639562] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.644661] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.651614] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.677680] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.703775] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.707363] Non-volatile memory driver v1.3 [ 2.708402] Linux agpgart interface v0.103 [ 2.733712] virtio_blk virtio1: [vda] 150688 512-byte logical blocks (77.2 MB/73.6 MiB) [ 2.735212] vda: detected capacity change from 0 to 77152256 [ 2.748067] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.750238] vdb: detected capacity change from 0 to 1073741824 [ 2.754569] libphy: Fixed MDIO Bus: probed [ 2.759173] usbcore: registered new interface driver usbserial_generic [ 2.761212] usbserial: USB Serial support registered for generic [ 2.764052] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.767469] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.768931] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.771204] mousedev: PS/2 mouse device common for all mice [ 2.773696] rtc_cmos 00:05: RTC can wake from S4 [ 2.776608] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.780895] rtc_cmos 00:05: registered as rtc0 [ 2.782016] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.784551] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.785109] intel_pstate: CPU model not supported [ 2.792736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.792912] hid: raw HID events driver (C) Jiri Kosina [ 2.798350] usbcore: registered new interface driver usbhid [ 2.800318] usbhid: USB HID core driver [ 2.801621] drop_monitor: Initializing network drop monitor service [ 2.803541] Initializing XFRM netlink socket [ 2.805386] NET: Registered protocol family 10 [ 2.807937] Segment Routing with IPv6 [ 2.808930] NET: Registered protocol family 17 [ 2.810492] mpls_gso: MPLS GSO support [ 2.814743] RAS: Correctable Errors collector initialized. [ 2.816404] AVX version of gcm_enc/dec engaged. [ 2.817466] AES CTR mode by8 optimization enabled [ 2.893814] sched_clock: Marking stable (2893744636, 0)->(3780419244, -886674608) [ 2.897390] registered taskstats version 1 [ 2.899714] Loading compiled-in X.509 certificates [ 2.902362] zswap: loaded using pool lzo/zbud [ 2.925297] Key type big_key registered [ 2.940328] Key type encrypted registered [ 2.941498] ima: No TPM chip found, activating TPM-bypass! [ 2.943178] ima: Allocated hash algorithm: sha1 [ 2.944568] ima: No architecture policies found [ 2.946393] evm: Initialising EVM extended attributes: [ 2.947674] evm: security.selinux [ 2.948925] evm: security.ima [ 2.950106] evm: security.capability [ 2.951369] evm: HMAC attrs: 0x1 [ 2.953425] rtc_cmos 00:05: setting system clock to 2026-09-30 12:46:45 UTC (1790772405) [ 2.957870] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 2.959837] debug: unmapping init [mem 0xffffffffb6782000-0xffffffffb6a58fff] [ 2.968148] Write protecting the kernel read-only data: 28672k [ 2.970237] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 2.972419] debug: unmapping init [mem 0xffffffffb5714000-0xffffffffb57fffff] [ 2.998797] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.004486] systemd[1]: Detected virtualization kvm. [ 3.006046] systemd[1]: Detected architecture x86-64. [ 3.007137] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.031330] systemd[1]: No hostname configured. [ 3.032356] systemd[1]: Set hostname to . [ 3.033840] random: systemd: uninitialized urandom read (16 bytes read) [ 3.035556] systemd[1]: Initializing machine ID from random generator. [ 3.080327] random: ln: uninitialized urandom read (6 bytes read) [ 3.149393] random: systemd: uninitialized urandom read (16 bytes read) [ 3.151443] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.155203] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.158149] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.768143] device-mapper: uevent: version 1.0.3 [ 3.770501] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.471926] virtio_net virtio0 ens2: renamed from eth0 [ 4.525054] scsi host0: ata_piix [ 4.546511] scsi host1: ata_piix [ 4.548081] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.550190] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.203917] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.464118] random: crng init done [ 9.465535] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.794652] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.987114] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.226527] SELinux: Disabled at runtime. [ 11.285883] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.292595] systemd[1]: Detected virtualization kvm. [ 11.294221] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.797034] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.800092] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.803895] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.807228] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.811335] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.820196] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.824504] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ 11.884682] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.309098] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.703227] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.756503] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.871085] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.882470] EDAC sbridge: Ver: 1.1.2 [ 14.058519] Key type dns_resolver registered [ 14.369057] NFS: Registering the id_resolver key type [ 14.370611] Key type id_resolver registered [ 14.372142] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-client login: [ 43.470373] libcfs: loading out-of-tree module taints kernel. [ 43.489841] Key type ._llcrypt registered [ 43.492633] Key type .llcrypt registered [ 43.883727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.892471] alg: No test for adler32 (adler32-zlib) [ 44.935952] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 45.300735] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 46.951270] Key type lgssc registered [ 47.711272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 50.588074] hrtimer: interrupt took 8063410 ns [ 186.514803] Lustre: Mounted lustre-client - version 2.17.59_4_ge4c1f18 [ 192.281044] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.150934] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing check_logdir /tmp/testlogs/ [ 209.684376] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing yml_node [ 212.447154] Lustre: lustre-OST0000-osc-ffff8a510b7cf800: disconnect after 23s idle [ 214.867028] Lustre: DEBUG MARKER: Client: 2.17.59.4 [ 217.553167] Lustre: DEBUG MARKER: MDS: 2.17.59.4 [ 220.079331] Lustre: DEBUG MARKER: OSS: 2.17.59.4 [ 221.999833] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 30 08:50:22 EDT 2026 [ 238.218939] Lustre: Unmounted lustre-client [ 274.777675] Key type lgssc unregistered [ 275.228631] LNet: 8379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.234581] LNetError: 8379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 275.259350] LNet: Removed LNI 192.168.201.45@tcp [ 276.641216] Key type .llcrypt unregistered [ 276.649153] Key type ._llcrypt unregistered [ 312.139106] Key type ._llcrypt registered [ 312.142231] Key type .llcrypt registered [ 312.471469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.507259] alg: No test for adler32 (adler32-zlib) [ 313.408386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 313.460448] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 313.475576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 317.638873] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 317.909901] LNet: 9283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.923478] LNetError: 9283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.949567] LNet: Removed LNI 192.168.201.45@tcp [ 318.762438] Key type .llcrypt unregistered [ 318.770017] Key type ._llcrypt unregistered [ 326.621970] Key type ._llcrypt registered [ 326.632223] Key type .llcrypt registered [ 328.324631] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:52:08 (1790772728) [ 328.802230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.849099] alg: No test for adler32 (adler32-zlib) [ 329.864714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.037630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.078705] LNet: 9693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.120332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.188291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.236856] LNet: 9700:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 338.552510] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:52:19 (1790772739) [ 338.714450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.743093] LNet: 10275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.767063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.880942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 346.385561] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:52:27 (1790772747) [ 346.611438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.650532] LNet: 10865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.673705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.743328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 354.481702] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:52:35 (1790772755) [ 354.618830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.656060] LNet: 11444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.697310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.861082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 362.522591] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:52:43 (1790772763) [ 362.627636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.668061] LNet: 12030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.684193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.812899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 363.555732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.202266] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:52:52 (1790772772) [ 371.301725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.343775] LNet: 12620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.413414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.560136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.124459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.867852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 380.723766] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:53:01 (1790772781) [ 380.868757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.926154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.000848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 387.339473] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:53:08 (1790772788) [ 387.510390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.547257] LNet: 13803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.555481] LNet: 13803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 387.638829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.729935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 387.783098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 388.233376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 388.271759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 388.722656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 388.800523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 389.604429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 389.658592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 390.399314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 390.441573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 391.011319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 391.071455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 391.574706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 391.594359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 392.219967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 392.283032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 392.956208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 393.030374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 393.689511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 393.729391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 394.348769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 394.372820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 394.831901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 394.879533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.423663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 395.459544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 402.542869] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:53:23 (1790772803) [ 402.772960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.852881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.975570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 403.001747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 410.397642] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:53:31 (1790772811) [ 410.739634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.780518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.892334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 410.970503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 418.323112] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:53:39 (1790772819) [ 418.426246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.483949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.584230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 418.619233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 426.575210] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:53:47 (1790772827) [ 426.676763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.748327] LNet: 16244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.755455] LNet: 16244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 426.798940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.985269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 427.021457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.348850] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:53:56 (1790772836) [ 435.469657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.502530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.576239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 435.630920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 442.730177] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:54:03 (1790772843) [ 442.822141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.878731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.006944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 443.049481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 451.824871] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:54:12 (1790772852) [ 452.024510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.131212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.291518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.853194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.650221] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:54:21 (1790772861) [ 460.985180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.044676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.375219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.880721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.539211] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:54:30 (1790772870) [ 469.718658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.783925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.888680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.087298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.062079] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:54:38 (1790772878) [ 477.199224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.277348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.944617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.165565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.508322] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:54:45 (1790772885) [ 484.660855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.765808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.866038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.904813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.357119] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:54:53 (1790772893) [ 493.528874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.574278] LNet: 21077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.585892] LNet: 21077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 493.629823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.814330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.875671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.079867] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:55:04 (1790772904) [ 504.323074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.449470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.734940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 513.773789] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:55:14 (1790772914) [ 514.136416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.229033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.341365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 522.518671] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:55:23 (1790772923) [ 522.613070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.702981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.818957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 523.426918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 532.768120] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:55:33 (1790772933) [ 532.905775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.969403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.095130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 533.155134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 541.051323] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:55:42 (1790772942) [ 541.251592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.326500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.455496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 541.517381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 548.131886] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:55:49 (1790772949) [ 548.230121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.294976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.395248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 548.455270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 554.534972] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:55:55 (1790772955) [ 554.673089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.758766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.912886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 554.956979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 561.864735] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:56:02 (1790772962) [ 561.944757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.013426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.100597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 562.179488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 562.208758] LNetError: 25810:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 562.253709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 562.281988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 568.597534] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:56:09 (1790772969) [ 568.748243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.815285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.937349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YdQB70SqGb [ 569.084943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YdQB70SqGb [ 569.173258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YdQB70SqGb [ 575.311677] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:56:16 (1790772976) [ 575.483374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.538410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.585684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 575.614679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 575.656049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 581.657806] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:56:22 (1790772982) [ 581.806598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.866295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.937880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 581.968368] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 583.368391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 584.492142] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 586.351915] Key type lgssc registered [ 587.723814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.941252] LNetError: 27571:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.45@tcp99 has no available nets [ 594.625473] Key type lgssc unregistered [ 594.965550] LNetError: 28686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.984529] LNet: Removed LNI 192.168.201.45@tcp [ 595.592563] Key type .llcrypt unregistered [ 595.595656] Key type ._llcrypt unregistered [ 604.592566] Key type ._llcrypt registered [ 604.595707] Key type .llcrypt registered [ 606.859444] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:56:46 (1790773006) [ 607.475818] Key type .llcrypt unregistered [ 607.483544] Key type ._llcrypt unregistered [ 615.867987] Key type ._llcrypt registered [ 615.869685] Key type .llcrypt registered [ 615.980476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 615.996246] alg: No test for adler32 (adler32-zlib) [ 617.047318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.209586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 617.231699] LNet: 29608:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 617.238628] LNet: 29608:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 617.243556] LNet: 29608:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 617.257747] LNet: Message forwarding will be enabled in 2s [ 620.324271] LNet: Message forwarding enabled [ 620.896757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.923387] LNet: Message forwarding will stop in 2s to 6s [ 624.430791] LNet: Message forwarding disabled [ 625.704943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.777196] LNet: 29707:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 625.804770] LNet: 29707:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 625.830599] LNet: 29707:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 625.859988] LNet: Message forwarding will be enabled in 2s [ 626.198205] LNet: 29720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.318362] Key type .llcrypt unregistered [ 627.323641] Key type ._llcrypt unregistered [ 644.448313] Key type ._llcrypt registered [ 644.455783] Key type .llcrypt registered [ 645.904774] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:57:26 (1790773046) [ 646.142813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.148969] alg: No test for adler32 (adler32-zlib) [ 647.091824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.174750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 654.198714] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:57:35 (1790773055) [ 654.422070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.462340] LNet: 31123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.502147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.610660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 654.645561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 654.675672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 654.709291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 654.753856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 654.788293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 654.834248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.865569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 654.911874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 654.964421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 654.992588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 655.016992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 655.050901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 655.134090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 655.166014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 655.219780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 655.261407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 655.286768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.316597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 655.338902] LNetError: 31167:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 662.496129] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:57:43 (1790773063) [ 662.646022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.704504] LNet: 31734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.739745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.851472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.880781] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:57:50 (1790773070) [ 671.763758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg145-server [ 674.192957] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:57:54 (1790773074) [ 674.397864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.443974] LNet: 32665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.480979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.563786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.584336] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 675.733459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[46-50]@tcp [ 675.762170] LNet: 32804:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 675.843191] LNetError: 32674:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 675.855330] LNetError: 32674:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.47@tcp has gone from up to down [ 683.129452] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:58:04 (1790773084) [ 685.515725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg145-server [ 687.265928] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:58:08 (1790773088) [ 687.547819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.569070] LNet: 33741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.575684] LNetError: 33741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.617415] LNet: Removed LNI 192.168.201.45@tcp [ 688.671434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.749103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.785724] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 689.662798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[46-51/2]@tcp [ 689.681803] LNet: 33858:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.689341] LNet: 33858:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 689.714494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[46-51/2]@tcp [ 689.727876] LNetError: 33745:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.48@tcp is being used as a gateway but routing feature is not turned on [ 689.753381] LNetError: 33745:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 689.774684] LNetError: 33745:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.48@tcp has gone from up to down [ 689.787108] LNetError: 33745:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 697.281931] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:58:18 (1790773098) [ 697.428364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.466762] LNet: 34449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.491349] LNetError: 34449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.527340] LNet: Removed LNI 192.168.201.45@tcp [ 697.572321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.711087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 697.761792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.815287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.911813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.001981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.050100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.129281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 698.208894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.241570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.318617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.422989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.475255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.556320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 698.677737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.723780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.799572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.881996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.949532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.006603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 699.114989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.152802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.248091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.364567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.426250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.493713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 699.639402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.710105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.784407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.841769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.881603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.960028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 700.056639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.090423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.139911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.663945] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:58:28 (1790773108) [ 710.244216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg145-server [ 712.458456] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:58:33 (1790773113) [ 714.490243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg145-server [ 716.549097] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:58:37 (1790773117) [ 716.726733] LNet: 35852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.730588] LNet: 35852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 717.442508] Key type .llcrypt unregistered [ 717.462713] Key type ._llcrypt unregistered [ 723.147678] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.322703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.333193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.342294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.977249] Key type ._llcrypt registered [ 725.978668] Key type .llcrypt registered [ 726.161657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.191448] alg: No test for adler32 (adler32-zlib) [ 727.085690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.131409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.157699] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 727.764681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.785653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.546561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.558929] LNet: Removed LNI 10.1.2.3@tcp [ 729.660768] LNet: 36570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.669112] LNetError: 36570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.691169] LNet: Removed LNI 192.168.201.45@tcp [ 730.429490] Key type .llcrypt unregistered [ 730.431738] Key type ._llcrypt unregistered [ 745.202788] Key type ._llcrypt registered [ 745.205071] Key type .llcrypt registered [ 746.776732] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:59:07 (1790773147) [ 747.345355] Key type .llcrypt unregistered [ 747.347921] Key type ._llcrypt unregistered [ 754.116371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.295267] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.301729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.310349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.065435] Key type ._llcrypt registered [ 757.068668] Key type .llcrypt registered [ 757.255897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.273282] alg: No test for adler32 (adler32-zlib) [ 758.177345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.227402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.246983] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 759.064502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.153759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.228794] LNet: 38066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.233484] LNetError: 38066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 761.237407] LNet: Removed LNI 10.1.2.3@tcp [ 761.244932] LNet: Removed LNI 192.168.201.45@tcp [ 761.876629] Key type .llcrypt unregistered [ 761.885496] Key type ._llcrypt unregistered [ 775.551777] Key type ._llcrypt registered [ 775.554055] Key type .llcrypt registered [ 776.991639] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:59:37 (1790773177) [ 777.493397] Key type .llcrypt unregistered [ 777.500242] Key type ._llcrypt unregistered [ 784.208608] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.449968] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.474754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.482379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.240437] Key type ._llcrypt registered [ 787.247717] Key type .llcrypt registered [ 787.542321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.577780] alg: No test for adler32 (adler32-zlib) [ 788.543517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.603604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 788.628837] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 791.232808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 791.276797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.931801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 793.969877] LNet: Removed LNI 10.1.2.3@tcp [ 793.986211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 794.014617] LNet: Removed LNI 10.1.2.31@tcp [ 794.032838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 794.075801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.283418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.317037] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.578640] LNet: 39981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.588331] LNetError: 39981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 796.594241] LNet: Removed LNI 10.1.2.31@tcp [ 796.602951] LNet: Removed LNI 10.1.2.3@tcp [ 797.240520] Key type .llcrypt unregistered [ 797.243596] Key type ._llcrypt unregistered [ 810.628380] Key type ._llcrypt registered [ 810.631712] Key type .llcrypt registered [ 811.953953] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:00:13 (1790773213) [ 812.565457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.579279] alg: No test for adler32 (adler32-zlib) [ 813.513284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.570688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.586069] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 814.356460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.375944] LNet: 40933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.383431] LNetError: 40933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.402870] LNet: Removed LNI 192.168.201.45@tcp [ 814.439542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.545964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.45@tcp [ 814.568813] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 815.616127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.45@tcp [ 815.647227] LNet: Removed LNI 192.168.201.45@tcp [ 815.706621] LNet: 41068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.447552] Key type .llcrypt unregistered [ 816.449373] Key type ._llcrypt unregistered [ 828.107402] Key type ._llcrypt registered [ 828.110446] Key type .llcrypt registered [ 829.301132] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:00:30 (1790773230) [ 830.995953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg145-server [ 832.616925] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:00:34 (1790773234) [ 833.170822] Key type .llcrypt unregistered [ 833.174168] Key type ._llcrypt unregistered [ 839.989419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.202180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.207062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.209436] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.736673] Key type ._llcrypt registered [ 842.738672] Key type .llcrypt registered [ 843.076433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.138120] alg: No test for adler32 (adler32-zlib) [ 844.129715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.238480] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 845.238251] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 847.531482] LNet: 42901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.537713] LNetError: 42901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.543659] LNet: Removed LNI 192.168.201.45@tcp1 [ 847.554443] LNet: Removed LNI 10.1.2.3@tcp2 [ 848.574473] Key type .llcrypt unregistered [ 848.583567] Key type ._llcrypt unregistered [ 863.787458] Key type ._llcrypt registered [ 863.789656] Key type .llcrypt registered [ 865.515376] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:01:06 (1790773266) [ 866.001269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.041731] alg: No test for adler32 (adler32-zlib) [ 867.019493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.077995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.105307] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 868.246146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.282892] LNet: 43883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.291349] LNetError: 43883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.311137] LNet: Removed LNI 192.168.201.45@tcp [ 868.332033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.420641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 868.458714] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 877.006325] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:01:17 (1790773277) [ 877.244282] LNet: 44475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.253271] LNetError: 44475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.267958] LNet: Removed LNI 192.168.201.45@tcp [ 878.067765] Key type .llcrypt unregistered [ 878.071819] Key type ._llcrypt unregistered [ 885.776862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 886.188187] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 886.190098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 886.206406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.835547] Key type ._llcrypt registered [ 888.837493] Key type .llcrypt registered [ 889.044030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 889.062231] alg: No test for adler32 (adler32-zlib) [ 889.991175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.039684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.061710] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 890.872657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 890.893972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.198516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.225652] LNet: 45217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.236410] LNetError: 45217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.240190] LNet: Removed LNI 10.1.2.3@tcp [ 893.261639] LNet: Removed LNI 192.168.201.45@tcp [ 893.321132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.442844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.483388] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 893.493147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.120555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.150663] LNet: 45244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.166321] LNetError: 45244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.173431] LNetError: 45244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 894.194451] LNet: Removed LNI 10.1.2.3@tcp [ 894.207454] LNet: Removed LNI 192.168.201.45@tcp [ 894.250372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.353135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 894.392044] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 894.401742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.120920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 895.137677] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 895.937420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.952965] LNet: 45406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.958938] LNetError: 45406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.965399] LNetError: 45406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 895.971607] LNet: Removed LNI 10.1.2.3@tcp [ 895.973703] LNet: Removed LNI 192.168.201.45@tcp [ 895.986430] LNet: Removed LNI 192.168.201.45@tcp2 [ 896.002861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.059991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 896.081659] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 896.086733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 896.092054] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 903.769217] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:01:44 (1790773304) [ 903.872355] LNet: 45998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.884942] LNetError: 45998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.895242] LNetError: 45998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 903.903644] LNet: Removed LNI 10.1.2.3@tcp [ 903.908717] LNet: Removed LNI 192.168.201.45@tcp [ 903.935656] LNet: Removed LNI 192.168.201.45@tcp2 [ 904.456598] Key type .llcrypt unregistered [ 904.459261] Key type ._llcrypt unregistered [ 912.270300] Key type ._llcrypt registered [ 912.272363] Key type .llcrypt registered [ 912.738226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.766079] alg: No test for adler32 (adler32-zlib) [ 913.855458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.943910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.967059] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 914.862591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.882839] LNet: 46400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.890819] LNetError: 46400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.903601] LNet: Removed LNI 192.168.201.45@tcp [ 914.923753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.997813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.033306] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 915.622986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.641500] LNet: 46430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.649429] LNetError: 46430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.661506] LNet: Removed LNI 192.168.201.45@tcp [ 915.689549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.745861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.776626] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 916.314540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.351028] LNet: Removed LNI 192.168.201.45@tcp [ 916.381959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.479380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.519793] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 916.905522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.198469] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.203081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.208168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.717260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.783269] LNet: 46538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.787422] LNet: 46538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 919.806968] LNetError: 46538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.818580] LNetError: 46538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.844271] LNet: Removed LNI 192.168.201.45@tcp [ 919.886417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.003695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 920.038963] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 928.298629] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:02:09 (1790773329) [ 928.671511] LNet: 47133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.708019] LNetError: 47133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.759811] LNet: Removed LNI 192.168.201.45@tcp [ 929.514430] Key type .llcrypt unregistered [ 929.516893] Key type ._llcrypt unregistered [ 938.834818] Key type ._llcrypt registered [ 938.841701] Key type .llcrypt registered [ 939.195838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.228381] alg: No test for adler32 (adler32-zlib) [ 940.165565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.212526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.240781] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 941.364041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.398582] LNet: 47533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.413373] LNetError: 47533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.442164] LNet: Removed LNI 192.168.201.45@tcp [ 941.465870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.579923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 941.632319] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 950.428990] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:02:31 (1790773351) [ 950.507485] LNet: 48118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.515745] LNetError: 48118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.533623] LNet: Removed LNI 192.168.201.45@tcp [ 951.442214] Key type .llcrypt unregistered [ 951.444615] Key type ._llcrypt unregistered [ 958.479624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.758868] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.762254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.773529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.347909] Key type ._llcrypt registered [ 961.350496] Key type .llcrypt registered [ 961.640293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.662563] alg: No test for adler32 (adler32-zlib) [ 962.664181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.734154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.774640] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 963.979558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.000611] LNet: 48575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.010664] LNetError: 48575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.032023] LNet: Removed LNI 192.168.201.45@tcp [ 964.049760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.136338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 964.167978] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 971.701493] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:02:52 (1790773372) [ 971.954124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.990804] LNet: 49174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.998700] LNetError: 49174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.039801] LNet: Removed LNI 192.168.201.45@tcp [ 972.073607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.178505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 979.191378] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:03:00 (1790773380) [ 979.458610] LNet: 49757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.143951] Key type .llcrypt unregistered [ 980.146363] Key type ._llcrypt unregistered [ 987.329458] Key type ._llcrypt registered [ 987.338182] Key type .llcrypt registered [ 987.610628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.636761] alg: No test for adler32 (adler32-zlib) [ 988.582647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.625325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.654687] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 989.832150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.889688] LNet: 50153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.897125] LNetError: 50153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.912036] LNet: Removed LNI 192.168.201.45@tcp [ 989.962335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.130366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 990.198712] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 997.734818] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:03:18 (1790773398) [ 997.918347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.960583] LNet: 50748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.966976] LNetError: 50748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.978310] LNet: Removed LNI 192.168.201.45@tcp [ 997.997882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.093455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1005.853246] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:03:26 (1790773406) [ 1007.366832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1009.044723] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:03:30 (1790773410) [ 1011.375660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1014.027457] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:03:34 (1790773414) [ 1016.018577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1018.097514] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:03:39 (1790773419) [ 1019.963974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1022.145519] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:03:42 (1790773422) [ 1024.131733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1026.028907] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:03:47 (1790773427) [ 1026.180379] LNet: 52764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.033698] Key type .llcrypt unregistered [ 1027.040903] Key type ._llcrypt unregistered [ 1034.445480] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1034.750108] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1034.762428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1034.764778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1037.451810] Key type ._llcrypt registered [ 1037.453763] Key type .llcrypt registered [ 1037.678180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.688924] alg: No test for adler32 (adler32-zlib) [ 1038.629388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.774829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.840770] LNet: Added LNI 192.168.201.45@tcp [8/512/0/180] [ 1038.852484] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1038.856270] LNet: 53091:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.941385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.957942] LNet: 53110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.962896] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.967353] LNet: Removed LNI 10.1.2.3@tcp [ 1038.970286] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.995881] LNet: Removed LNI 192.168.201.45@tcp [ 1039.016861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.098496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1039.139445] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1039.267220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.315343] LNet: 53134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.327893] LNetError: 53134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.336921] LNet: Removed LNI 192.168.201.45@tcp [ 1039.395591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.520833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1039.540434] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1039.543429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1039.546102] LNet: 53142:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1039.602286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.631425] LNet: 53160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.646245] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1039.652197] LNet: Removed LNI 10.1.2.3@tcp [ 1039.656776] LNetError: 53160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.684368] LNet: Removed LNI 192.168.201.45@tcp [ 1039.738560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.875239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1039.911427] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1039.916275] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1039.920228] LNet: 53169:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1040.125916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.170742] LNet: 53195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.186524] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.203191] LNet: Removed LNI 10.1.2.3@tcp [ 1040.204845] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.224638] LNet: Removed LNI 192.168.201.45@tcp [ 1040.239520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.319366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1040.347649] LNet: Added LNI 192.168.201.45@tcp [8/256/0/182] [ 1040.352145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1040.479619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.500192] LNet: 53223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.507848] LNetError: 53223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.514854] LNet: Removed LNI 10.1.2.3@tcp [ 1040.519278] LNetError: 53223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.540927] LNet: Removed LNI 192.168.201.45@tcp [ 1040.582808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.655045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1040.671602] LNet: Added LNI 192.168.201.45@tcp [8/256/0/182] [ 1040.676107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1040.678223] LNet: 53231:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1040.782407] LNet: Removed LNI 10.1.2.3@tcp [ 1040.822907] LNet: Removed LNI 192.168.201.45@tcp [ 1041.726251] Key type .llcrypt unregistered [ 1041.728694] Key type ._llcrypt unregistered [ 1054.436950] Key type ._llcrypt registered [ 1054.439392] Key type .llcrypt registered [ 1056.116457] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:04:16 (1790773456) [ 1057.941876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1059.931109] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:04:20 (1790773460) [ 1060.398818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.430915] alg: No test for adler32 (adler32-zlib) [ 1061.319064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.387468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.408327] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1062.556877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.45 [ 1062.585995] LNet: 54494:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1062.596969] LNet: Removed LNI 192.168.201.45@tcp [ 1068.732749] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:04:29 (1790773469) [ 1068.860211] LNet: 55058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.867498] LNetError: 55058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.907225] LNet: Removed LNI 192.168.201.45@tcp [ 1069.547596] Key type .llcrypt unregistered [ 1069.550727] Key type ._llcrypt unregistered [ 1078.188213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1078.586428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.214363] Key type ._llcrypt registered [ 1080.216496] Key type .llcrypt registered [ 1080.355437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.383455] alg: No test for adler32 (adler32-zlib) [ 1081.814493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.787026] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:04:49 (1790773489) [ 1089.090600] LNet: 55970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.110678] LNetError: 55970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1089.126955] LNet: Removed LNI 10.1.2.3@tcp [ 1089.763775] Key type .llcrypt unregistered [ 1089.767634] Key type ._llcrypt unregistered [ 1096.870722] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1097.306499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1097.313891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1097.318294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1100.305102] Key type ._llcrypt registered [ 1100.308781] Key type .llcrypt registered [ 1100.590737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.638639] alg: No test for adler32 (adler32-zlib) [ 1101.626030] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1108.999584] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:05:09 (1790773509) [ 1110.672438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1112.984896] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:05:13 (1790773513) [ 1113.150230] LNet: 57148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.172170] LNetError: 57148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1113.206294] LNet: Removed LNI 10.1.2.3@tcp [ 1113.911627] Key type .llcrypt unregistered [ 1113.913192] Key type ._llcrypt unregistered [ 1121.150179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.448952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.453291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.468668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1124.268888] Key type ._llcrypt registered [ 1124.275592] Key type .llcrypt registered [ 1124.544227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.587689] alg: No test for adler32 (adler32-zlib) [ 1125.554957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.651120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1136.922358] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:05:37 (1790773537) [ 1137.521698] LNet: 58388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.530238] LNetError: 58388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1137.549944] LNet: Removed LNI 10.1.2.3@tcp [ 1138.377272] Key type .llcrypt unregistered [ 1138.378978] Key type ._llcrypt unregistered [ 1145.858567] Key type ._llcrypt registered [ 1145.865913] Key type .llcrypt registered [ 1146.112466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.137229] alg: No test for adler32 (adler32-zlib) [ 1147.209762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1147.238142] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1157.050192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1157.982792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 1158.021145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 1158.056935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1158.100838] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1158.445122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.515570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.561180] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1158.568662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.624199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.653613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1158.706216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.764780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.832573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.864801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.887477] LNet: 58828:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.907849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.947101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.019916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.056329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.096328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.146269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.179786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.880426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 1159.910530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 1159.955907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1159.978118] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1160.248410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.285287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.312526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.361484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.389575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1160.438870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.486521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.543166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.584953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.604906] LNet: 58940:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.623893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.656405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.683558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.712148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.739883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.764886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.794588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.366642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 1161.398512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 1161.426942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1161.446159] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1161.451628] LNet: Skipped 1 previous similar message [ 1161.725635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.757735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.786941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.816597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.857833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1161.904612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.960625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.008251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.074432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.094804] LNet: 59051:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.114554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.144421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.170490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.207956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.243807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.278510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.308386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.923823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 1162.961630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 1163.003350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1163.310770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.350977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.394616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.449787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.543407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1163.581126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.611537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.647619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.677514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.713787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.738226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.764807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.787626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.809606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.840720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.867329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.544238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 1164.584795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 1164.661577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1164.700684] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1164.708206] LNet: Skipped 1 previous similar message [ 1164.942953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.987467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.015177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.038717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.065702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1165.097955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.151603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.185510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.223715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.260235] LNet: 59273:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.265475] LNet: 59273:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1165.288765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.323274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.411706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.450663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.518114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.569454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.610958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1166.369834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 1166.432130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 1166.497912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1166.860730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.888838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.921601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.956925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.985420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1167.019461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.093836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.146180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.205196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.267332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.296957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.334981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.369355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.405335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.436931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.481794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.725478] LNet: 59426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.730933] LNetError: 59426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.808781] LNet: Removed LNI 192.168.201.45@tcp [ 1169.247819] Key type .llcrypt unregistered [ 1169.250344] Key type ._llcrypt unregistered [ 1188.232973] Key type ._llcrypt registered [ 1188.235039] Key type .llcrypt registered [ 1189.735021] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:06:30 (1790773590) [ 1190.636372] Key type .llcrypt unregistered [ 1190.637928] Key type ._llcrypt unregistered [ 1198.663426] Key type ._llcrypt registered [ 1198.666906] Key type .llcrypt registered [ 1198.892419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.910910] alg: No test for adler32 (adler32-zlib) [ 1200.035946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.055460] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1208.740482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1220.576219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.646966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1220.672478] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1221.798849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1222.475975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 1222.507134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 1 [ 1222.535464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 1222.567838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 1 [ 1222.609385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 1222.657445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 1 [ 1222.715630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 1222.779824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 1 [ 1222.835908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1222.878051] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1222.886514] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp1 [ 1223.136760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.168586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.206707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.269020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.354083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1223.440992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.474106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.520453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.581175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.624911] LNet: 60900:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.662190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.705982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.756202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.806557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.847397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.897922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.907615] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1223.919906] LNet: Skipped 3 previous similar messages [ 1223.946430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.668241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 1224.711412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 1 [ 1224.778156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 1224.910771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 1 [ 1224.962208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 1224.994637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 1 [ 1225.030663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 1225.064590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 1 [ 1225.097773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1225.119422] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1225.361895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.391822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.421300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.449680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.478819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1225.502931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.527422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.551816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.581693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.598620] LNet: 61023:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.614321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.640993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.671530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.699555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.725283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.755777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.792915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.279579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 1226.339728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 1 [ 1226.402967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 1226.480271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 1 [ 1226.531710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 1226.571763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 1 [ 1226.619397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 1226.665374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 1 [ 1226.710559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1227.056325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.088992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.135740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.163709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.196592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1227.225754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.255565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.282825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.323117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.345149] LNet: 61146:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.378421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.414945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.515904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.563744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.627717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.670894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.721454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.474574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 1228.512549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 1 [ 1228.552630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 1228.596557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 1 [ 1228.659945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 1228.703987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 1 [ 1228.734559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 1228.768413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 1 [ 1228.799949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1228.814449] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1228.817407] LNet: Skipped 5 previous similar messages [ 1229.085111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.117144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.142789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.181463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.224374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1229.266562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.302868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.338475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.389402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.412735] LNet: 61270:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.437429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.467718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.505991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.548271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.607532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.644301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.687345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.274740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 1230.314524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 1 [ 1230.366505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 1230.476580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 1 [ 1230.545974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 1230.581329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 1 [ 1230.622354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 1230.655610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 1 [ 1230.719946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1230.966254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.000273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.031989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.065599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.110677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1231.141563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.177225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.210659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.256286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.294994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.330456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.360921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.389878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.423690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.455413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.487985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.061137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 1232.109259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_error -r 1 [ 1232.155873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 1232.214583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_error -r 1 [ 1232.261618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 1232.295456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_error -r 1 [ 1232.333206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 1232.366590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_error -r 1 [ 1232.401196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1232.582412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.611664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.640377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.671810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.705817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1232.729498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.763285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.801701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.868621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.899526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.931143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.965740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.005639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.044970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.078978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.108398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.591558] LNet: 61592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.602462] LNetError: 61592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.612378] LNet: Removed LNI 192.168.201.45@tcp [ 1240.677824] LNet: Removed LNI 192.168.201.45@tcp1 [ 1241.049929] Key type .llcrypt unregistered [ 1241.054860] Key type ._llcrypt unregistered [ 1259.720899] Key type ._llcrypt registered [ 1259.722362] Key type .llcrypt registered [ 1261.339255] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:07:42 (1790773662) [ 1262.243542] Key type .llcrypt unregistered [ 1262.246913] Key type ._llcrypt unregistered [ 1269.533452] Key type ._llcrypt registered [ 1269.534984] Key type .llcrypt registered [ 1269.811647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.865630] alg: No test for adler32 (adler32-zlib) [ 1270.816547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.866248] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1279.077922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1279.828964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 1279.904975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 1279.942949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1279.968721] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1280.326508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.351570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.386183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.415720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.446843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.493099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.542201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.584638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.637886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.651730] LNet: 62842:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.672993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.717205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.746701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.782567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.829996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.892358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.920700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.369356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 1281.390713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 1281.420117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1281.440117] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1281.673803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.711856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.743549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.770525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.800811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1281.828965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.859664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.885211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.907702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.920395] LNet: 62953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.932147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.954976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.977658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.003322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.027091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.057621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.085163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.685191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 1282.724796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 1282.756530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1282.770636] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1283.047810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.099607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.184422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.261567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.332209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1283.424835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.463236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.497658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.538328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.559076] LNet: 63064:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.579990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.614744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.661073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.702364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.739582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.773949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.812437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1284.109418] LNet: 63106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.117911] LNetError: 63106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.159400] LNet: Removed LNI 192.168.201.45@tcp [ 1285.692162] Key type .llcrypt unregistered [ 1285.695187] Key type ._llcrypt unregistered [ 1305.506407] Key type ._llcrypt registered [ 1305.509381] Key type .llcrypt registered [ 1307.107942] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:08:27 (1790773707) [ 1307.934814] Key type .llcrypt unregistered [ 1307.937095] Key type ._llcrypt unregistered [ 1315.346358] Key type ._llcrypt registered [ 1315.348606] Key type .llcrypt registered [ 1315.630289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1315.652651] alg: No test for adler32 (adler32-zlib) [ 1316.743774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1316.808114] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1326.004409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1336.830772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.898784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.923920] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1338.010962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1338.665484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 1338.722834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 1 [ 1338.766828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 1338.827697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 1 [ 1338.875360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 1338.939090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 1 [ 1338.984645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 1339.037053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 1 [ 1339.080627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1339.102928] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1339.109612] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp1 [ 1339.366066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1339.400810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1339.430675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1339.452143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1339.488816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1339.518390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.549141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.578767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.612297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.629533] LNet: 64579:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.650993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.685158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.719862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.753918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.792567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.859674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.909851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.967884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1340.612399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 1340.646786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_error -r 1 [ 1340.687440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 1340.750439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_error -r 1 [ 1340.805734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 1340.864381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_error -r 1 [ 1340.909648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 1340.943904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_error -r 1 [ 1340.980343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1340.996456] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1341.001262] LNet: Skipped 2 previous similar messages [ 1341.288661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.324172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1341.352137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.373951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.397617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1341.425802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.451302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.476489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.504926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.521768] LNet: 64705:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.539156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.576868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.604786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.634682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.662851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.691728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.719452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.747773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.266622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 1342.305473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_timeout -r 1 [ 1342.334677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 1342.384428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_timeout -r 1 [ 1342.436377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 1342.477549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_timeout -r 1 [ 1342.508463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 1342.544439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_timeout -r 1 [ 1342.577659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 1342.601138] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1342.876571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.905502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.940496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.971828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.000401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1343.032672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.060352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.094034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.131625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.151993] LNet: 64830:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.206611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.302431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.361981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.402987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.430651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.471217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.515622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1343.574609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.997941] LNet: 64908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.006483] LNetError: 64908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.013321] LNet: Removed LNI 192.168.201.45@tcp [ 1349.021254] LNet: Removed LNI 192.168.201.45@tcp1 [ 1349.508776] Key type .llcrypt unregistered [ 1349.511301] Key type ._llcrypt unregistered [ 1367.674231] Key type ._llcrypt registered [ 1367.675655] Key type .llcrypt registered [ 1369.338558] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:09:30 (1790773770) [ 1370.983667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1372.476555] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:09:33 (1790773773) [ 1373.183337] Key type .llcrypt unregistered [ 1373.185202] Key type ._llcrypt unregistered [ 1378.202543] Key type ._llcrypt registered [ 1378.203881] Key type .llcrypt registered [ 1378.466935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1378.488648] alg: No test for adler32 (adler32-zlib) [ 1379.432955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1379.467839] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1388.401952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1389.434690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1389.473588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1389.513645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1389.534824] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1389.780604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1389.792717] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1389.810325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1389.843730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1389.890867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1389.930292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1389.982855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1390.019603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1390.041950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1390.080912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1390.115473] LNet: 66454:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1390.146926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1390.205792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1390.259408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1390.302975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1390.367291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1390.418408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1390.910947] LNet: 66493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.917092] LNetError: 66493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.959602] LNet: Removed LNI 192.168.201.45@tcp [ 1391.469765] Key type .llcrypt unregistered [ 1391.472477] Key type ._llcrypt unregistered [ 1412.958893] Key type ._llcrypt registered [ 1412.964171] Key type .llcrypt registered [ 1413.185747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1413.206548] alg: No test for adler32 (adler32-zlib) [ 1414.233544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1414.272728] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1422.900560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1433.697298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.743045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1433.770371] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1434.810384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1435.476620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1435.510203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e network_timeout -r 1 [ 1435.537493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1435.573479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e network_timeout -r 1 [ 1435.609395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1435.642858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e network_timeout -r 1 [ 1435.671706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1435.698996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e network_timeout -r 1 [ 1435.720508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1435.734507] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1435.807325] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp1 [ 1435.816449] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp1 [ 1435.950718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1435.978464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1436.000613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1436.027439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1436.052617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1436.078769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1436.105730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1436.126869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1436.151815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1436.166506] LNet: 67389:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1436.184913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1436.217198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1436.241774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1436.259499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1436.285396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1436.315852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1441.498689] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1442.048545] LNet: 67463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.069913] LNetError: 67463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.081370] LNet: Removed LNI 192.168.201.45@tcp [ 1442.103802] LNet: Removed LNI 192.168.201.45@tcp1 [ 1442.749898] Key type .llcrypt unregistered [ 1442.757153] Key type ._llcrypt unregistered [ 1459.377479] Key type ._llcrypt registered [ 1459.379216] Key type .llcrypt registered [ 1461.388818] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:11:01 (1790773861) [ 1461.945074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1461.957793] alg: No test for adler32 (adler32-zlib) [ 1462.980767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.070869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.093966] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1463.969547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.992968] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1464.907481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1464.976885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1465.025746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1465.069090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1465.111774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1465.138627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1465.154977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1465.164851] LNetError: 68562:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1466.079454] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1466.091199] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1468.131033] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1468.972037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.223488] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1472.680421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.747615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1472.796458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.808663] LNet: 68669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.812825] LNetError: 68669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.822361] LNet: Removed LNI 192.168.201.45@tcp [ 1472.846383] LNet: Removed LNI 192.168.201.45@tcp1 [ 1472.879845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.967291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.019824] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1474.113181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.185945] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1475.743509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1475.836776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1475.931256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1476.011751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1476.036500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1476.092597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1476.135346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1476.172902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1476.194154] LNetError: 68941:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1477.089293] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1479.734508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1483.231518] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1483.238053] LNet: Skipped 1 previous similar message [ 1487.721982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1487.742956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.771511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1497.246288] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:11:36 (1790773896) [ 1497.349067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.383960] LNet: 69654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.394398] LNetError: 69654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.405360] LNetError: 69654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1497.420352] LNet: Removed LNI 192.168.201.45@tcp [ 1497.448992] LNet: Removed LNI 192.168.201.45@tcp1 [ 1497.507128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.602774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.651410] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1498.862466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1498.891265] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1500.351644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1500.429577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1500.502187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1500.574985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1500.624456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1500.663664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1500.692285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 0 [ 1500.837801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.669737] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.45@tcp [ 1501.671959] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1501.695488] LNet: Skipped 1 previous similar message [ 1515.561618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1515.776995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1515.808639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1515.838626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 500 [ 1515.967298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.605434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.652664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.674779] LNet: 70137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.680524] LNetError: 70137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.687599] LNetError: 70137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1521.700107] LNet: Removed LNI 192.168.201.45@tcp [ 1521.717055] LNet: Removed LNI 192.168.201.45@tcp1 [ 1521.732189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.775709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1521.802989] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1522.544837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1522.574744] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1523.430842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1523.458620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1523.510212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1523.531898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1523.570449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1523.639756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1523.684835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1523.734492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 0 [ 1524.834634] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1524.841075] LNet: Skipped 3 previous similar messages [ 1525.084172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1534.376874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1534.430710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1534.464313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.611225] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:12:22 (1790773942) [ 1542.128939] LNet: 71174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.133222] LNetError: 71174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.141357] LNetError: 71174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1542.145859] LNet: Removed LNI 192.168.201.45@tcp [ 1542.156827] LNet: Removed LNI 192.168.201.45@tcp1 [ 1542.935308] Key type .llcrypt unregistered [ 1542.937343] Key type ._llcrypt unregistered [ 1548.056206] Key type ._llcrypt registered [ 1548.057796] Key type .llcrypt registered [ 1548.178321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1548.207430] alg: No test for adler32 (adler32-zlib) [ 1549.076777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1549.106653] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1557.226217] Lustre: DEBUG MARKER: Initial discovery [ 1557.794632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1561.484494] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1561.993206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1562.058254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1562.114830] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1562.154122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1562.215485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.145@tcp -r 1 -m GET -l 3 [ 1562.273869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.145@tcp -r 1 -m GET -e local_error [ 1562.317279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.145@tcp -r 1 -m PUT -l 6 [ 1562.463836] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1564.488988] Lustre: DEBUG MARKER: Force oleg145-client.virtnet to discover 192.168.201.145@tcp (in background) [ 1564.716839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1566.026865] Lustre: DEBUG MARKER: Force oleg145-server to discover 192.168.201.45@tcp [ 1567.779257] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1568.773684] Lustre: DEBUG MARKER: Wait for 71934 [ 1570.425984] Lustre: DEBUG MARKER: Finished wait on 71934 [ 1570.750637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1570.782066] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1570.789793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1570.967970] LNet: 72385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.972133] LNetError: 72385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.984178] LNet: Removed LNI 192.168.201.45@tcp [ 1571.259480] Key type .llcrypt unregistered [ 1571.261050] Key type ._llcrypt unregistered [ 1589.634264] Key type ._llcrypt registered [ 1589.635638] Key type .llcrypt registered [ 1591.062694] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:13:12 (1790773992) [ 1591.924430] Key type .llcrypt unregistered [ 1591.926821] Key type ._llcrypt unregistered [ 1600.114349] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1600.532549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1600.543897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1600.562646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1603.040335] Key type ._llcrypt registered [ 1603.041642] Key type .llcrypt registered [ 1603.116985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1603.122863] alg: No test for adler32 (adler32-zlib) [ 1604.036325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.083929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.152671] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1604.942928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1604.959862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1614.086736] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:13:35 (1790774015) [ 1614.344457] LNet: 74472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.349981] LNetError: 74472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1614.353346] LNet: Removed LNI 10.1.2.3@tcp [ 1614.360485] LNet: Removed LNI 192.168.201.45@tcp [ 1615.229567] Key type .llcrypt unregistered [ 1615.234390] Key type ._llcrypt unregistered [ 1622.627763] Key type ._llcrypt registered [ 1622.630704] Key type .llcrypt registered [ 1622.941371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1622.980229] alg: No test for adler32 (adler32-zlib) [ 1623.888266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.915890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.929144] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1624.344351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1624.360094] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1624.964466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.45@tcp [ 1626.015407] LNetError: Unexpected error -22 connecting to 192.168.201.45@tcp at host 192.168.201.45:988 [ 1634.399368] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:13:55 (1790774035) [ 1634.782816] LNet: 75613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.788595] LNetError: 75613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1634.791419] LNet: Removed LNI 10.1.2.3@tcp [ 1634.806810] LNet: Removed LNI 192.168.201.45@tcp [ 1635.256538] Key type .llcrypt unregistered [ 1635.257645] Key type ._llcrypt unregistered [ 1640.748371] Key type ._llcrypt registered [ 1640.750720] Key type .llcrypt registered [ 1640.850068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.868829] alg: No test for adler32 (adler32-zlib) [ 1641.812069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.900065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.959732] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1642.679748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1642.713269] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1643.607331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.45@tcp1 --nid 192.168.201.45@tcp2 [ 1648.547270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp1 192.168.201.45@tcp2 [ 1648.571579] LNetError: 76448:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.45@tcp1 to 192.168.201.45@tcp2 - Route table may be misconfigured [ 1648.590511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp2 192.168.201.45@tcp1 [ 1654.582861] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:14:15 (1790774055) [ 1654.815970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.849879] LNet: 77021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.860828] LNetError: 77021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.871331] LNet: Removed LNI 192.168.201.45@tcp1 [ 1655.915788] LNet: Removed LNI 192.168.201.45@tcp2 [ 1655.932907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.003893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.024811] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1656.875672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.891583] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1657.697943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1657.727488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.45@tcp [ 1657.767940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp1 192.168.201.45@tcp1 [ 1657.816468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp -d 192.168.201.45@tcp -e network_timeout [ 1657.850462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -e network_timeout [ 1657.884247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -e network_timeout [ 1657.911281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -e network_timeout [ 1657.941485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1657.980649] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1658.001795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1658.017961] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1658.109428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1658.203924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1664.329387] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:14:25 (1790774065) [ 1664.576187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.598689] LNet: 77882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.608905] LNetError: 77882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.615238] LNetError: 77882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.622046] LNet: Removed LNI 192.168.201.45@tcp [ 1664.639523] LNet: Removed LNI 192.168.201.45@tcp1 [ 1664.690632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.767184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1665.453085] Key type .llcrypt unregistered [ 1665.456056] Key type ._llcrypt unregistered [ 1677.621875] Key type ._llcrypt registered [ 1677.624136] Key type .llcrypt registered [ 1679.241226] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:14:40 (1790774080) [ 1679.837200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1679.855974] alg: No test for adler32 (adler32-zlib) [ 1680.959797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.772939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1684.160389] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:14:45 (1790774085) [ 1684.532907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.567761] LNet: 79018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.597174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.674269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.693243] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1685.309808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.336331] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1686.283534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1686.317835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp1 [ 1686.357679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp1 [ 1693.044349] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:14:54 (1790774094) [ 1694.685899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg145-server [ 1696.310521] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:14:57 (1790774097) [ 1698.411726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg145-server [ 1700.165645] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:15:01 (1790774101) [ 1702.150092] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg145-server [ 1703.962155] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:15:05 (1790774105) [ 1706.019430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg145-server [ 1707.888497] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:15:09 (1790774109) [ 1710.004279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg145-server [ 1711.818368] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:15:12 (1790774112) [ 1714.101989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg145-server [ 1716.184477] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:15:17 (1790774117) [ 1718.171684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg145-server [ 1720.057448] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:15:21 (1790774121) [ 1722.236409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg145-server [ 1724.125461] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:15:25 (1790774125) [ 1726.446413] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg145-server [ 1728.185413] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:15:29 (1790774129) [ 1728.398284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.431242] LNet: 82897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.437558] LNetError: 82897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.444209] LNet: Removed LNI 192.168.201.45@tcp [ 1729.515964] LNet: Removed LNI 192.168.201.45@tcp1 [ 1729.559132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.634724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.664106] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1730.422077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1730.502271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1730.794845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.818926] LNet: 83044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.824316] LNetError: 83044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.834477] LNetError: 83044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1730.856637] LNet: Removed LNI 192.168.201.45@tcp [ 1730.878206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.921382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1730.941411] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1731.728652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1731.961948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.005534] LNetError: 83190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.019590] LNet: Removed LNI 192.168.201.45@tcp [ 1732.032375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.082885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.102786] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1732.786095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1732.875614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1733.047862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.068117] LNet: 83337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.075777] LNet: 83337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1734.130444] LNet: Removed LNI 192.168.201.45@tcp [ 1734.154272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.215152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1734.236801] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1735.043227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1735.304445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.318349] LNetError: 83482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.321327] LNetError: 83482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1735.339832] LNet: Removed LNI 192.168.201.45@tcp [ 1735.363110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.436628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.459657] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1736.222771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1736.307218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1737.563478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.592895] LNet: 83630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.601443] LNet: 83630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1738.662491] LNet: Removed LNI 192.168.201.45@tcp [ 1738.705945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.796895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1738.851261] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1739.822421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1740.092530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.117467] LNetError: 83775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.122531] LNetError: 83775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1741.157863] LNet: Removed LNI 192.168.201.45@tcp [ 1741.225579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.324659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.377261] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1742.254704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1742.337096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1742.682788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.716814] LNet: Removed LNI 192.168.201.45@tcp [ 1743.759381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.822527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1743.857797] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1744.704716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1744.871468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.899945] LNet: Removed LNI 192.168.201.45@tcp [ 1745.936126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.015639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.042560] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1746.986381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1747.065328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1747.365750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.400492] LNet: 84216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.406949] LNet: 84216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1748.460316] LNet: Removed LNI 192.168.201.45@tcp [ 1748.496272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.548910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1748.572919] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1749.537584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1751.012751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.044826] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.051301] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1751.070283] LNet: Removed LNI 192.168.201.45@tcp [ 1751.099528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.172362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.199905] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1752.105301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1752.226880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1752.618159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.771867] LNet: Removed LNI 192.168.201.45@tcp [ 1753.800433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.912462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1753.965277] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1754.962749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1756.490581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.549962] LNet: Removed LNI 192.168.201.45@tcp [ 1757.585425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.673719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.708728] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1758.497821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1758.592856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.631142] LNet: Removed LNI 192.168.201.45@tcp [ 1758.688611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.746144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.776843] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1759.590196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1759.673554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.709247] LNet: Removed LNI 192.168.201.45@tcp [ 1759.734618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.826104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1759.923656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.964648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.020890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1760.122799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.158038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.260403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1760.377387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.414727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.461406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1760.478074] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1761.417622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.443667] LNet: Removed LNI 192.168.201.45@tcp [ 1761.466033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.536422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1769.284934] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:16:10 (1790774170) [ 1769.500076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.521301] LNet: 85694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.525264] LNet: 85694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1769.553294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.623995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1769.639473] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1771.042995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.059663] LNetError: 85832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.064916] LNetError: 85832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1771.078239] LNet: Removed LNI 192.168.201.45@tcp231 [ 1771.099199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.183576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1771.218874] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1771.226084] LNet: 85840:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1771.683410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1771.743408] LNet: Removed LNI 192.168.201.45@tcp231 [ 1771.763619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1771.798977] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1772.795286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.861154] LNet: Removed LNI 192.168.201.45@tcp231 [ 1772.881409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.960326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.990043] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/0] [ 1772.997301] LNet: 85993:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.618379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1773.668258] LNet: Removed LNI 192.168.201.45@tcp231 [ 1773.702292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1773.761117] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/0] [ 1774.871422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.906620] LNet: Removed LNI 192.168.201.45@tcp231 [ 1774.924970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.023639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1775.050230] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/60] [ 1775.064988] LNet: 86144:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.529877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1775.557378] LNet: Removed LNI 192.168.201.45@tcp231 [ 1775.571327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1775.591271] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/60] [ 1776.739593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.768909] LNet: Removed LNI 192.168.201.45@tcp231 [ 1776.798606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.867549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1776.888746] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1777.271735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1777.315551] LNet: Removed LNI 192.168.201.45@tcp231 [ 1777.336277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1777.376624] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1786.362552] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:16:27 (1790774187) [ 1786.574525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.645064] LNet: Removed LNI 192.168.201.45@tcp231 [ 1786.662105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.732996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1786.801509] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1787.716079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.803747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.848302] LNet: Removed LNI 192.168.201.45@tcp232 [ 1787.873130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1787.912716] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1789.023651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1789.109994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1789.172685] LNet: Removed LNI 192.168.201.45@tcp232 [ 1789.194263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1789.225312] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1789.822798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1789.853512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1789.886517] LNet: Removed LNI 192.168.201.45@tcp232 [ 1789.921733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1789.949642] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1790.600887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1790.639283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1790.670670] LNet: Removed LNI 192.168.201.45@tcp232 [ 1790.688529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1797.503622] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:16:38 (1790774198) [ 1798.476687] Key type .llcrypt unregistered [ 1798.478855] Key type ._llcrypt unregistered [ 1805.262827] Key type ._llcrypt registered [ 1805.265055] Key type .llcrypt registered [ 1805.419993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1805.437123] alg: No test for adler32 (adler32-zlib) [ 1806.447970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1806.472692] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1814.727925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1823.888746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.967353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1824.010774] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1825.050589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1825.824740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 2 [ 1825.876471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 2 [ 1825.909630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 2 [ 1825.955538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 2 [ 1826.023841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_interrupt -r 2 [ 1826.079230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 2 [ 1826.128937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_interrupt -r 2 [ 1826.192930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 2 [ 1826.246408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1826.281211] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1826.302854] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1826.692239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1826.756025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1826.807238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1826.883303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1826.942477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1827.002586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.065273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.122899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.175078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.194114] LNet: 88803:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.215674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1827.291941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1827.343992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.414359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1827.497271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1827.567486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.066654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 2 [ 1828.123704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 2 [ 1828.169689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 2 [ 1828.208537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 2 [ 1828.235877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_dropped -r 2 [ 1828.268058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 2 [ 1828.305587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_dropped -r 2 [ 1828.345257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 2 [ 1828.419544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1828.459544] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1828.763457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.842571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1828.895359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.941771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1828.990982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1829.020845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.066086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.089897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.133083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.168839] LNet: 88926:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.199949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.258352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.327904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.360643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.414925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.461532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1830.186788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 2 [ 1830.257926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 2 [ 1830.322482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 2 [ 1830.369520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 2 [ 1830.392983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_aborted -r 2 [ 1830.437287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 2 [ 1830.505583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_aborted -r 2 [ 1830.614290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 2 [ 1830.668946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1830.714267] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1831.073326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1831.111211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1831.162301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.196844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.230219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1831.273616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.294316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.334185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.381808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.425393] LNet: 89048:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.458987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.515396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.577986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.640942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.711434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.756646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.515192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 2 [ 1832.587135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 2 [ 1832.641893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 2 [ 1832.703601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 2 [ 1832.769961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_no_route -r 2 [ 1832.874639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 2 [ 1832.928605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_no_route -r 2 [ 1832.980218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 2 [ 1833.048062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1833.102544] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1833.368135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.397487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1833.415972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.432268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.445897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1833.465138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.482770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.540892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.575778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.595528] LNet: 89170:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.612903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.632744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.655716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.745302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.815507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.877322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1834.790150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 2 [ 1834.874263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 2 [ 1834.958987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 2 [ 1835.013456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 2 [ 1835.114565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_timeout -r 2 [ 1835.178814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 2 [ 1835.237764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_timeout -r 2 [ 1835.315120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 2 [ 1835.395718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1835.867758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.901808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1835.933160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1836.033464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1836.100811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1836.167487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1836.212397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1836.296495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1836.341396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1836.424689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.494222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.566610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.678368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.766430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.836516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.898250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 2 [ 1837.968239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 2 [ 1838.030293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 2 [ 1838.100076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 2 [ 1838.233610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_dropped -r 2 [ 1838.373302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 2 [ 1838.478446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_dropped -r 2 [ 1838.536374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 2 [ 1838.579712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1838.616049] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1838.620347] LNet: Skipped 1 previous similar message [ 1838.630352] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 1839.129776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.157453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1839.185445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.228946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.282188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1839.342347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.377166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.463829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.515333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.542960] LNet: 89415:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1839.552676] LNet: 89415:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1839.582082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.649183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.724706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.791554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1839.853929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1839.938804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1847.791415] LNet: 89490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.798790] LNetError: 89490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.808093] LNet: Removed LNI 192.168.201.45@tcp [ 1847.819350] LNet: Removed LNI 192.168.201.45@tcp1 [ 1848.320870] Key type .llcrypt unregistered [ 1848.326990] Key type ._llcrypt unregistered [ 1866.049344] Key type ._llcrypt registered [ 1866.052874] Key type .llcrypt registered [ 1867.216671] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:17:48 (1790774268) [ 1867.644973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.664521] alg: No test for adler32 (adler32-zlib) [ 1868.612132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.686920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.719585] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1869.584477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1869.629809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1869.648418] LNetError: 90469:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1869.669675] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1869.724946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.45@tcp [ 1869.844432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1876.453495] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:17:57 (1790774277) [ 1876.609300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.631328] LNet: 91055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.658252] LNetError: 91055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.681267] LNet: Removed LNI 192.168.201.45@tcp [ 1876.735866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.791372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1876.814324] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1877.507680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1877.545249] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1878.397265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.45@tcp [ 1878.415643] LNetError: 91325:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1878.817897] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1879.671951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.45@tcp [ 1887.544556] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:18:08 (1790774288) [ 1889.369763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg145-server [ 1890.832692] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:18:12 (1790774292) [ 1892.733635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg145-server [ 1894.466359] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:18:15 (1790774295) [ 1896.134451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg145-server [ 1897.669718] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:18:18 (1790774298) [ 1897.938085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.964255] LNet: 92948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.970625] LNetError: 92948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.978049] LNet: Removed LNI 192.168.201.45@tcp [ 1898.980985] LNetError: 92948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.015527] LNet: Removed LNI 192.168.201.45@tcp1 [ 1899.062811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.125105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1899.160099] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1899.850630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1899.942710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1899.984742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1900.110097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1900.172378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1900.295956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1906.778615] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:18:27 (1790774307) [ 1906.873265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.886133] LNet: 93672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.891514] LNetError: 93672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.915413] LNet: Removed LNI 192.168.201.45@tcp [ 1906.929577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.981406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.000358] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1914.034570] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:18:35 (1790774315) [ 1914.171827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.194223] LNet: 94381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.197775] LNetError: 94381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.232525] LNet: Removed LNI 192.168.201.45@tcp [ 1914.250330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.303797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.339190] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1921.252576] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:18:42 (1790774322) [ 1922.869422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1924.674449] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:18:45 (1790774325) [ 1925.200465] LNet: 95417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.205689] LNetError: 95417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.219979] LNet: Removed LNI 192.168.201.45@tcp [ 1925.818512] Key type .llcrypt unregistered [ 1925.822376] Key type ._llcrypt unregistered [ 1932.039937] Key type ._llcrypt registered [ 1932.043287] Key type .llcrypt registered [ 1932.187247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.207423] alg: No test for adler32 (adler32-zlib) [ 1933.143127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1933.169331] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1940.994106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1945.621403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.145@tcp [ 1945.924410] LNet: 95834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.931385] LNetError: 95834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.940948] LNet: Removed LNI 192.168.201.45@tcp [ 1946.337784] Key type .llcrypt unregistered [ 1946.340333] Key type ._llcrypt unregistered [ 1957.164118] Key type ._llcrypt registered [ 1957.166655] Key type .llcrypt registered [ 1958.902349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1960.585136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1962.652651] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:19:23 (1790774363) [ 1964.105111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1966.372993] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:19:27 (1790774367) [ 1968.517940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg145-server [ 1971.393005] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:19:31 (1790774371) [ 1974.327064] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg145-server [ 1978.004860] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:19:37 (1790774377) [ 1980.685200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.699459] alg: No test for adler32 (adler32-zlib) [ 1981.856919] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1982.053445] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1983.727238] Key type lgssc registered [ 1984.718402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.572944] Key type lgssc unregistered [ 2006.219925] LNet: 99016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.227498] LNetError: 99016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.247648] LNet: Removed LNI 192.168.201.45@tcp [ 2007.706579] Key type .llcrypt unregistered [ 2007.708579] Key type ._llcrypt unregistered [ 2024.336720] Key type ._llcrypt registered [ 2024.340910] Key type .llcrypt registered [ 2026.409409] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:20:26 (1790774426) [ 2026.788229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.808594] alg: No test for adler32 (adler32-zlib) [ 2027.807481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.939403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.005659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.069565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.156071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.158330] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:20:35 (1790774435) [ 2035.320657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.391556] LNet: 100434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.454708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.596674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.614615] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2036.460924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.483939] LNet: 100583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.491946] LNetError: 100583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.503134] LNet: Removed LNI 192.168.201.45@tcp [ 2036.521969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.583193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2036.602791] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2037.011376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.027946] LNetError: 100611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.046992] LNet: Removed LNI 192.168.201.45@tcp [ 2037.095666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.150669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2037.185277] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2045.253920] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:20:46 (1790774446) [ 2045.398271] LNet: 101194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.410171] LNet: 101194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2045.419448] LNetError: 101194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.438328] LNet: Removed LNI 192.168.201.45@tcp [ 2046.332576] Key type .llcrypt unregistered [ 2046.334983] Key type ._llcrypt unregistered [ 2053.482502] Key type ._llcrypt registered [ 2053.487777] Key type .llcrypt registered [ 2053.664564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2053.686189] alg: No test for adler32 (adler32-zlib) [ 2054.623754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.739135] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2054.743526] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.752798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2055.785686] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.789074] LNet: Skipped 2 previous similar messages [ 2055.794450] Ping reply received with mapping: [ 2055.797423] found metadata reply NI[192.168.201.45@tcp] [ 2055.801797] 2d c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.804966] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.810626] 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.966367] LNet: 101598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.972498] LNetError: 101598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.997741] LNet: Removed LNI 192.168.201.45@tcp [ 2057.865662] Key type .llcrypt unregistered [ 2057.869971] Key type ._llcrypt unregistered [ 2069.783415] Key type ._llcrypt registered [ 2069.785438] Key type .llcrypt registered [ 2071.295272] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:21:12 (1790774472) [ 2072.144413] Key type .llcrypt unregistered [ 2072.148284] Key type ._llcrypt unregistered [ 2079.459192] Key type ._llcrypt registered [ 2079.463225] Key type .llcrypt registered [ 2079.652930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.673100] alg: No test for adler32 (adler32-zlib) [ 2080.838460] LNetError: 102639:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2081.275480] Key type .llcrypt unregistered [ 2081.278728] Key type ._llcrypt unregistered [ 2086.917572] Key type ._llcrypt registered [ 2086.924337] Key type .llcrypt registered [ 2088.629800] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:21:29 (1790774489) [ 2089.214498] Key type .llcrypt unregistered [ 2089.216145] Key type ._llcrypt unregistered [ 2097.102372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2097.112724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2099.763638] Key type ._llcrypt registered [ 2099.765451] Key type .llcrypt registered [ 2100.001892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2100.033940] alg: No test for adler32 (adler32-zlib) [ 2101.074411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.201130] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2110.876300] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:21:51 (1790774511) [ 2111.174464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.211066] LNet: 104418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.222949] LNetError: 104418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2111.237115] LNet: Removed LNI 10.1.2.3@tcp [ 2111.282221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.163783] Key type .llcrypt unregistered [ 2112.165904] Key type ._llcrypt unregistered [ 2119.268086] Key type ._llcrypt registered [ 2119.269689] Key type .llcrypt registered [ 2119.454782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.466631] alg: No test for adler32 (adler32-zlib) [ 2120.386729] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2120.433807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2120.555455] LNet: 104705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.560205] LNetError: 104705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.571860] LNet: Removed LNI 192.168.201.45@tcp [ 2121.165076] Key type .llcrypt unregistered [ 2121.169165] Key type ._llcrypt unregistered [ 2135.442943] Key type ._llcrypt registered [ 2135.447088] Key type .llcrypt registered [ 2137.063788] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:22:17 (1790774537) [ 2138.108169] Key type .llcrypt unregistered [ 2138.110125] Key type ._llcrypt unregistered [ 2146.376917] Key type ._llcrypt registered [ 2146.383699] Key type .llcrypt registered [ 2146.704989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2146.721206] alg: No test for adler32 (adler32-zlib) [ 2154.697747] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:22:35 (1790774555) [ 2154.999615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.053383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.136267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.168035] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2156.085811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.119578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2156.149719] LNet: Removed LNI 192.168.201.45@tcp [ 2156.437674] LNet: 106448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.903411] Key type .llcrypt unregistered [ 2156.907726] Key type ._llcrypt unregistered [ 2170.278592] Key type ._llcrypt registered [ 2170.284366] Key type .llcrypt registered [ 2171.649802] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:22:52 (1790774572) [ 2172.301768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.318921] alg: No test for adler32 (adler32-zlib) [ 2173.258040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.320324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.383076] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2174.146227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2174.173578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.45@tcp [ 2174.197563] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2174.202325] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2174.207552] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2174, 0, 0 [ 2174.217580] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2174.224696] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2174.232592] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2174.240827] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2174, 0, 0 [ 2174.249446] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2174.251255] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2174.259242] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2174.271609] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2174, 0, 2224 [ 2174.289986] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2174.299857] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2174.306448] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2174.316410] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2174, 0, 2224 [ 2174.328296] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2180.846906] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:23:01 (1790774581) [ 2181.396919] LNet: 107986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.401924] LNetError: 107986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.450480] LNet: Removed LNI 192.168.201.45@tcp [ 2183.287470] Key type .llcrypt unregistered [ 2183.293107] Key type ._llcrypt unregistered [ 2190.871846] Key type ._llcrypt registered [ 2190.874880] Key type .llcrypt registered [ 2191.291105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2191.315059] alg: No test for adler32 (adler32-zlib) [ 2192.257842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2192.284383] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2201.209466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 2212.594718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.651281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2212.679344] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 2213.801546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 2213.978495] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2214.266384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2214.268751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2214.275990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2216.686794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2216.712817] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2228.420426] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:23:49 (1790774629) [ 2228.627588] LNet: 109499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.633248] LNetError: 109499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.647194] LNet: Removed LNI 192.168.201.45@tcp [ 2229.663229] LNet: Removed LNI 192.168.201.45@tcp1 [ 2229.671430] LNetError: 109499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2229.686797] LNetError: 109499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2229.716721] LNet: Removed LNI 10.1.2.3@tcp99 [ 2230.444741] Key type .llcrypt unregistered [ 2230.447314] Key type ._llcrypt unregistered [ 2238.093428] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2238.097837] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2240.548744] Key type ._llcrypt registered [ 2240.553508] Key type .llcrypt registered [ 2240.751739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2240.762510] alg: No test for adler32 (adler32-zlib) [ 2241.696041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.795976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.817784] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2242.764697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2242.789804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.293652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2245.337135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.382151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2245.396320] LNetError: 110244:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2245.437293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.467427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2245.817424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.885792] LNet: 110264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.891278] LNetError: 110264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2245.896521] LNet: Removed LNI 10.1.2.3@tcp [ 2246.950490] LNet: Removed LNI 192.168.201.45@tcp [ 2247.713402] Key type .llcrypt unregistered [ 2247.714847] Key type ._llcrypt unregistered [ 2260.991722] Key type ._llcrypt registered [ 2260.993775] Key type .llcrypt registered [ 2262.747310] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:24:23 (1790774663) [ 2263.067702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.084738] alg: No test for adler32 (adler32-zlib) [ 2263.989208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.050612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.103717] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2265.076527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2265.171197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg145-client@tcp [ 2272.508318] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:24:33 (1790774673) [ 2272.646810] LNet: 111790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.660804] LNetError: 111790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.684947] LNet: Removed LNI 192.168.201.45@tcp [ 2273.361710] Key type .llcrypt unregistered [ 2273.363977] Key type ._llcrypt unregistered [ 2279.774880] Key type ._llcrypt registered [ 2279.777941] Key type .llcrypt registered [ 2280.059639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.094577] alg: No test for adler32 (adler32-zlib) [ 2281.059055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2281.091185] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2281.125624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2289.036462] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:24:49 (1790774689) [ 2290.620545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2292.907777] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:24:53 (1790774693) [ 2294.824254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2296.870476] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:24:57 (1790774697) [ 2297.005147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.040326] LNet: 113217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.045313] LNetError: 113217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.060762] LNet: Removed LNI 192.168.201.45@tcp [ 2297.098479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.171218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.199495] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2298.172482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2298.267758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2306.667107] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:25:07 (1790774707) [ 2306.778199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.798914] LNet: 113925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.807415] LNetError: 113925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.823187] LNet: Removed LNI 192.168.201.45@tcp [ 2306.887095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.980101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2307.031457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2307.967737] Key type .llcrypt unregistered [ 2307.970211] Key type ._llcrypt unregistered [ 2321.939329] Key type ._llcrypt registered [ 2321.941384] Key type .llcrypt registered [ 2323.716487] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:25:24 (1790774724) [ 2324.222303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.249593] alg: No test for adler32 (adler32-zlib) [ 2325.307990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.400287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.453057] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2326.430669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2326.538296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2333.358417] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:25:34 (1790774734) [ 2333.640311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.662359] LNet: 115488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.672754] LNetError: 115488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.688742] LNet: Removed LNI 192.168.201.45@tcp [ 2333.710794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.771953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2333.829262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2339.947177] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:25:41 (1790774741) [ 2340.036926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.075796] LNet: 116061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.092302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.143982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2340.628290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2340.648162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2340.711914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2341.219478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2341.263783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2341.341622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.389674] LNet: 116102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.422758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.556230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2342.120373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2342.153420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2342.225359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2342.721636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2342.757262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2349.547493] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:25:50 (1790774750) [ 2349.638675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.678859] LNet: 116693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.738080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.826404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.899867] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2350.873962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.906899] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2351.785382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2351.836410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2352.084270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.101298] LNetError: 116982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.119305] LNet: Removed LNI 192.168.201.45@tcp [ 2353.122394] LNetError: 116982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.144167] LNet: Removed LNI 192.168.201.45@tcp2 [ 2353.181678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.247694] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2353.252472] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2353.258422] LNet: 116988:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.804575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2354.084978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.103892] LNet: 117025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.108451] LNet: 117025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2355.167732] LNet: Removed LNI 192.168.201.45@tcp [ 2355.170238] LNetError: 117025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.184442] LNetError: 117025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2355.209069] LNet: Removed LNI 192.168.201.45@tcp2 [ 2355.234187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.291731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.321505] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2355.965082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.983042] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2356.882207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2356.911991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2356.963447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2357.238230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.303185] LNet: Removed LNI 192.168.201.45@tcp [ 2359.334792] LNet: Removed LNI 192.168.201.45@tcp2 [ 2359.392973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.487372] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2359.507815] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2359.514834] LNet: 117316:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2360.245703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2360.439769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.463798] LNetError: 117353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.468866] LNetError: 117353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2360.479505] LNet: Removed LNI 192.168.201.45@tcp [ 2361.515773] LNet: Removed LNI 192.168.201.45@tcp2 [ 2361.554336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.635233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2361.663553] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2362.665885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2362.685162] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2363.651738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2363.761577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2363.995217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.038148] LNet: 117633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.048751] LNet: 117633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2365.087952] LNet: Removed LNI 192.168.201.45@tcp [ 2365.102097] LNet: Removed LNI 192.168.201.45@tcp2 [ 2365.160874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.287439] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2365.306974] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2365.314823] LNet: 117639:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.043667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2366.297658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.331236] LNet: Removed LNI 192.168.201.45@tcp [ 2367.343095] LNet: Removed LNI 192.168.201.45@tcp2 [ 2367.405809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.524314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.565227] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2368.231899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2368.246872] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2369.250454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2369.316720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2369.379689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2369.609327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.636642] LNetError: 117961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.640307] LNetError: 117961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2369.647712] LNet: Removed LNI 192.168.201.45@tcp [ 2370.726812] LNet: Removed LNI 192.168.201.45@tcp2 [ 2370.781080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2370.903729] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2370.911548] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2370.915860] LNet: 117968:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2371.604947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2371.908918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.955612] LNet: Removed LNI 192.168.201.45@tcp [ 2372.974898] LNet: Removed LNI 192.168.201.45@tcp2 [ 2373.033131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.105172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2373.132730] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2374.229897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2374.263533] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2375.322886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2375.376559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2375.468878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2376.037931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.119677] LNet: Removed LNI 192.168.201.45@tcp [ 2377.155039] LNet: Removed LNI 192.168.201.45@tcp2 [ 2377.232575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.325430] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2377.332529] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2377.336496] LNet: 118309:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2378.112398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2378.663672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.743190] LNet: Removed LNI 192.168.201.45@tcp [ 2379.764549] LNet: Removed LNI 192.168.201.45@tcp2 [ 2379.792065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.871743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2379.897470] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2380.705452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2380.727831] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2381.771715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2381.816928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2381.854808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2381.909934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2382.254933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.271672] LNet: 118663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.274803] LNet: 118663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2383.327243] LNet: Removed LNI 192.168.201.45@tcp [ 2384.369052] LNet: Removed LNI 192.168.201.45@tcp2 [ 2384.436911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.507288] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2384.511994] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2385.149808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2392.298166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2393.837702] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:26:35 (1790774795) [ 2393.973544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.041765] LNetError: 119425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.055857] LNetError: 119425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2394.068903] LNet: Removed LNI 192.168.201.45@tcp [ 2395.115764] LNet: Removed LNI 192.168.201.45@tcp2 [ 2395.141267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.432935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2395.463795] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2396.205153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2396.230935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2397.194352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.212284] LNet: Removed LNI 10.1.2.3@tcp [ 2397.218217] LNet: Removed LNI 192.168.201.45@tcp [ 2397.235186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.287524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2397.308764] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2397.311034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2397.667693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2397.691693] LNet: Removed LNI 192.168.201.45@tcp [ 2397.711628] LNet: Removed LNI 10.1.2.3@tcp [ 2403.781968] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:26:44 (1790774804) [ 2403.978766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2404.049369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.135622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.217768] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2404.984412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2405.020308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2405.056426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.094904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.129555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.153553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.183796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.241497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.307529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.334586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.373918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.428753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2405.650657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2411.754775] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 09:26:53 (1790774813) [ 2411.911141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.031197] LNet: Removed LNI 192.168.201.45@tcp [ 2413.066525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.116527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.144556] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2413.911113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2413.944195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2413.969237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2413.993842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.017797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.049257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.081144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.115713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.154184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.181774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.220694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.253326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.364822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.414451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.467212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.513953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.575393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.641106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.689955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.754931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.804214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.866964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2414.904837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 2422.079731] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:27:03 (1790774823) [ 2422.212078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.255302] LNet: 121794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.264320] LNet: 121794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2423.342160] LNet: Removed LNI 192.168.201.45@tcp [ 2423.373929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.643987] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2423.902873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2423.912953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2423.924209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2426.503766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2426.549923] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2427.606742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2427.646482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2431.550745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2431.685580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2431.831740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.45@tcp -m PUT -e local_timeout -r 1 [ 2431.966835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2432.098030] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2432.103537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2432.562001] LNetError: 121865:0:(lib-move.c:4794:lnet_parse()) 192.168.201.45@tcp, src 192.168.201.45@tcp: Dropping PUT (error -108 looking up sender) [ 2433.576919] LNetError: 122323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2433.603341] LNetError: 122323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2433.636255] LNet: Removed LNI 10.1.2.3@tcp [ 2434.669449] LNet: Removed LNI 192.168.201.45@tcp [ 2435.263789] Key type .llcrypt unregistered [ 2435.267276] Key type ._llcrypt unregistered [ 2449.459332] Key type ._llcrypt registered [ 2449.460938] Key type .llcrypt registered [ 2450.410880] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:27:31 (1790774851) [ 2451.498834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2451.538095] alg: No test for adler32 (adler32-zlib) [ 2452.446633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2452.831431] LNet: 123175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.547488] Key type .llcrypt unregistered [ 2453.549277] Key type ._llcrypt unregistered [ 2463.059295] Key type ._llcrypt registered [ 2463.065155] Key type .llcrypt registered [ 2463.666693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2463.737872] alg: No test for adler32 (adler32-zlib) [ 2464.783478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2464.822532] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2473.299121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 2473.583990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2474.491677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.145@tcp [ 2474.917945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2474.976317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2475.023174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2475.068835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2475.104233] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 2475.119831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2475.197145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2475.275996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2475.336696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2475.382641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2475.449538] LNet: 123626:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2475.477543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2475.527378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2475.572394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2475.595202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2475.631296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2475.660726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2476.246961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2476.499905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2476.526804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2476.570938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2476.606388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2476.637869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2476.670832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2476.695897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2476.728943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2476.766855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2476.798564] LNet: 123746:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2476.823958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2476.856104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2476.881391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2476.910927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2476.939587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2476.980106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2477.606373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.145@tcp [ 2477.827520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2477.849598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2477.869575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2477.895731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2477.923675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2477.953383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.977421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.999678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2478.025327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2478.042934] LNet: 123865:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.065309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2478.099173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2478.149961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2478.192137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2478.230262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2478.256689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.839300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2479.048197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.071677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.099840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.158273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.189925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2479.244973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.298642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.333271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.374493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.417511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2479.453929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2479.489362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2479.523175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2479.587243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2479.647315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2479.996967] LNet: 124030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.005515] LNetError: 124030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2481.072666] LNet: Removed LNI 192.168.201.45@tcp [ 2481.526173] Key type .llcrypt unregistered [ 2481.528867] Key type ._llcrypt unregistered [ 2501.309569] Key type ._llcrypt registered [ 2501.313178] Key type .llcrypt registered [ 2502.665938] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:28:23 (1790774903) [ 2503.169292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.189624] alg: No test for adler32 (adler32-zlib) [ 2504.172052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.569753] LNet: 124917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.297030] Key type .llcrypt unregistered [ 2505.299622] Key type ._llcrypt unregistered [ 2512.757696] Key type ._llcrypt registered [ 2512.759910] Key type .llcrypt registered [ 2512.915959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2512.931470] alg: No test for adler32 (adler32-zlib) [ 2513.885673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2513.911445] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2521.808929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 2531.029296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2531.093326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2531.118501] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 2531.983983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 2532.143841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2532.712780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.145@tcp [ 2532.932950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.955674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2532.981198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2533.003928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2533.037803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2533.078613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2533.106276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2533.137245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2533.179284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2533.196979] LNet: 125575:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2533.215572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2533.258856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2533.299109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2533.345945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2533.401375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2533.407300] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 2533.452551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.236413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2534.542801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2534.567326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.584503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.606981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.639873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2534.666789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.690554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.714741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.738582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.763717] LNet: 125694:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.789226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.825526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.883370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.911874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.941760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.971268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.620778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.145@tcp [ 2535.876542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2535.902650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2535.928224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2535.951142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2535.975932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2536.015474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2536.038866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2536.065307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2536.088284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2536.101684] LNet: 125813:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2536.123869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2536.158248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2536.187335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2536.203415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2536.222318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2536.250404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2536.847397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2537.027679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2537.047492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2537.074185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2537.103136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2537.135845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2537.159200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2537.184939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2537.217528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2537.252871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2537.283599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2537.333271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2537.365976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2537.388142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2537.418528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2537.441323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2541.937256] LNet: 126013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2541.946669] LNetError: 126013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2541.952037] LNet: Removed LNI 192.168.201.45@tcp [ 2541.971897] LNet: Removed LNI 192.168.201.45@tcp1 [ 2542.320977] Key type .llcrypt unregistered [ 2542.324666] Key type ._llcrypt unregistered [ 2557.885881] Key type ._llcrypt registered [ 2557.888237] Key type .llcrypt registered [ 2559.034604] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:29:20 (1790774960) [ 2559.426948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2559.438623] alg: No test for adler32 (adler32-zlib) [ 2560.368572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2560.396172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2560.425933] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2561.284395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2561.304622] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 2562.062833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 2562.130376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2563.167166] LNet: Removed LNI 192.168.201.45@tcp [ 2569.037611] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:29:30 (1790774970) [ 2570.503530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2572.456552] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:29:33 (1790774973) [ 2580.240159] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:29:41 (1790774981) [ 2587.142140] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:29:48 (1790774988) [ 2596.254804] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:29:57 (1790774997) [ 2604.321497] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:30:05 (1790775005) [ 2611.710326] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:30:12 (1790775012) [ 2619.196852] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:30:20 (1790775020) [ 2627.136571] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:30:28 (1790775028) [ 2634.790234] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:30:36 (1790775036) [ 2642.183640] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:30:43 (1790775043) [ 2649.357562] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:30:50 (1790775050) [ 2658.173290] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:30:59 (1790775059) [ 2666.790685] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:31:07 (1790775067) [ 2674.963241] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:31:16 (1790775076) [ 2683.338371] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:31:24 (1790775084) [ 2692.024609] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:31:32 (1790775092) [ 2701.901467] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:31:42 (1790775102) [ 2710.239811] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:31:51 (1790775111) [ 2710.416961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2710.459362] LNet: 137710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.464154] LNetError: 137710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.474768] LNet: Removed LNI 192.168.201.45@tcp2 [ 2710.489080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2710.600202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2710.617050] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2711.507541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2711.536946] LNet: 137852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2711.543109] LNetError: 137852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2711.559295] LNet: Removed LNI 192.168.201.45@tcp [ 2711.575505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2711.647810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2711.665996] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2718.177449] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:31:59 (1790775119) [ 2726.491624] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:32:07 (1790775127) [ 2735.523878] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:32:16 (1790775136) [ 2743.943070] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:32:25 (1790775145) [ 2749.971742] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:32:31 (1790775151) [ 2757.135281] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:32:38 (1790775158) [ 2764.010936] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:32:45 (1790775165) [ 2772.859807] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:32:53 (1790775173) [ 2783.035187] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:33:03 (1790775183) [ 2791.681068] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:33:12 (1790775192) [ 2801.346477] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:33:21 (1790775201) [ 2811.824455] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:33:32 (1790775212) [ 2821.300746] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2598 sec ========== 09:33:42 (1790775222) [ 2822.300235] LNet: 145319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.304934] LNetError: 145319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.342474] LNet: Removed LNI 192.168.201.45@tcp [ 2823.166471] Key type .llcrypt unregistered [ 2823.168215] Key type ._llcrypt unregistered [ 2837.098753] Key type ._llcrypt registered [ 2837.100462] Key type .llcrypt registered [ 2837.417617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2837.428546] alg: No test for adler32 (adler32-zlib) [ 2838.466155] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2838.700476] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 2840.295282] Key type lgssc registered [ 2841.988081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.626890] Lustre: Mounted lustre-client - version 2.17.59_4_ge4c1f18 [ 2929.085303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2937.171871] Lustre: Unmounted lustre-client [ 2964.205493] Key type lgssc unregistered [ 2964.561086] LNet: 148283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2964.593454] LNetError: 148283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2964.627238] LNet: Removed LNI 192.168.201.45@tcp [ 2965.731248] Key type .llcrypt unregistered [ 2965.733699] Key type ._llcrypt unregistered